Releases: HubSpot/Singularity
Releases · HubSpot/Singularity
Singularity 0.20.0
Changes in 0.20.0
Check out the 0.20.0 milestone to see new features / bugfixes in detail.
Configuration Changes
#1784 simplified the configuration for the weighting of different resources when evaluating offers. An old version of the config would look like:
mesos:
longRunningFreeResourceWeight: 0.5
longRunningUsedResourceWeight: 0.5
nonLonRunningFreeResourceWeight: 0.5
nonLongRunningUsedResourceWeight: 0.5
scoringStrategy: SPREAD_TASK_USAGE
longRunningUsedCpuWeightForOffer: 0.25
longRunningUsedMemWeightForOffer: 0.65
longRunningUsedDiskWeightForOffer: 0.1
freeCpuWeightForOffer: 0.25
freeMemWeightForOffer: 0.65
freeDiskWeightForOffer: 0.1
defaultOfferScoreForMissingUsage: 0.3
maxNonLongRunningUsedResourceWeight: 0.5
considerNonLongRunningTaskLongRunningAfterRunningForSeconds: 3600The new simplified config is now (defaults are shown):
mesos:
allocatedResourceWeight: 0.5
inUseResourceWeight: 0.5
cpuWeight: 0.4
memWeight: 0.4
diskWeight: 0.2Default behavior remains the same while eliminating complexity in the scoring system for evaluating offers
New Features
- Singularity now has new api docs (PR link) powered by an updated version of swagger and using open api 3.0. A page is available in the UI when running Singularity to view the docs for the release you are currently running.
- 1775 - Add an optional cpu hard limit
Improvements
- 1800 - Ability to maintain the same path for custom nav bar links
- 1795 - Add clear all buttons for dead slaves and inactive hosts
- 1783 - Simplify mesos master uri parsing
- 1796 - Allow a buffer for tasks near the cpu hard limit
- 1727 - Allow user to configure which load metric is used for offer scoring
- 1720 - Account for expected usage when scoring offers
- 1763 - Support SSE S3 in SingularityUploader
- 1770 - Support a configurable delay for task shuffles.
- 1776 - Configurably omit offers from hosts that are overloaded
- 1787 - Add run time column to request item's task history
- 1769 - Support placeholders in webhook URIs.
- 1778 - Include current task usage in the task shuffle cleanup messages
- 1785 - Send email on failing replacement tasks.
- 1788 - Add ability to set attributes that mark slave for only preemptible tasks
Bug Fixes
- 1805 - Fix race condition where two tasks decommission at the same time
- 1797 - Fix UI message for incremental deploy counts
- 1716 - Remove expiring scale when new scale has no expiration
- 1724 - Uploader refactoring and additional attempts for immediate uploaders
- 1732 - Better check for finish of a bounce
- 1781 - Log the full list of healthy task ids when killing a task
- 1777 - Do not upload files outside task sandbox
- 1782 - Proxy run-nows to the leader.
- 1773 - Retry lost tasks
- 1794 - Key run-nows with runId in addition to current epoch millis.
Thanks
Singularity 0.19.2
Changes in 0.19.2
Check out the 0.19.2 milestone to see new features / bugfixes in detail.
Improvements
- 1762 - Allow deploy of paused requests
- 1758 - Enable framework auth over http api
- 1754 - Add an offer scoring mode accounting for max historical usage of all tasks
- 1760 - Sort tasks to shuffle by overusage, not usage
Bug Fixes
- 1761 - Remove task credit ui calls, no longer used
- 1762 - Don't require group overrides for s3 logs listing + allow log level override in docker setup
- 1759 - Fix duration in logging statement
- 1757 - Don't count pending requests towards underprovisioning
- 1741 - Fix lag banner in UI
- 1755 - Catch exceptions in offer scoring and log them appropriately
- 1748 - Permalink request within group
- 1751 - Limit the number of tasks that can shuffle for cpu rebalance at once
- 1752 - Clarified command not found exception
- 1766 - More usage collection in parallel, less webhooks in parallel
- 1767 - Periodically flush the queue to make sure batch work does not get stuck
- 1768 - Keep track of offers not accounted for in SingularityOfferHolders returned by checkOffers
Thanks
Singularity 0.19.1
Changes in 0.19.1
This is a bugfix release
Check out the 0.19.1 milestone to see bugfixes in detail.
- 1740 - Only send healthcheck object from ui when setting all fields
Singularity 0.17.1
Changes in 0.17.1
This is a bugfix release
Check out the 0.17.1 milestone to see bugfixes in detail.
- 1740 - Only send healthcheck object from ui when setting all fields
Singularity 0.19.0
Changes in 0.19.0
Check out the 0.19.0 milestone to see new features / bugfixes in detail.
New Features
- 1668/1650 Adds support for more overrides in run now requests. You can now override items like environment variables for individual runs of a SingularityRequest by
POSTing json in the form of aSingularityRunNowRequest - 1690 - Adds initial support for Mesos containers, with volume sources and network mapping. These can be specified
in thecontainerInfo.volumesandcontainerInfo.mesos.imagesections of theSingularityContainerInfoin yourSingularityDeploy - 1702 - Updates the internal locking scheme for Singularity to allow more parallel processing. As a result, the concurrency of offer and status update processing can now be tuned, with increased concurrency coming with a cost of increased memory/cpu usage by the scheduler. The following parameters in the
mesossection of theSingularityConfigurationimpact concurrency and tuning of the scheduler:statusUpdateConcurrencyLimit- The number of status updates that can be processed in parallel. Defaults to 500 and is backed by its own cached thread poolmaxStatusUpdateQueueSize- A semaphore limits the number of submissions to the status update cached thread pool. If there are currently no more permits available (i.e. >statusUpdateConcurrencyLimitstatus updates), these are added to a queue where they wait until more capacity is available. This configuration parameter controls the max size of that queue. It is recommended that this be set a bit above the maximum number of tasks you expect to have active in Singularity at any one time, due to the fact that during reconciliation a status update for each task is sent in rapid succession.offersConcurrencyLimit- The number of offer scoring calculations and checks to be done in parallel. Defaults to 100. This should generally not need to be updated.
Improvements
API/Scheduler
- 1666 - Set AVAILABILITY_ZONE on the default task environment.
- 1681 - Provide option to prevent emails for scale events
- 1653 - Endpoint to allow users to delete pending on-demand tasks
- 1657 - Include disk resources when scoring offers
- 1683 - Support usage of DefaultServerFactory to find port
- 1692 - Wire up overrides for the S3 uploader path.
- 1616 - Remove swagger jackson version override
- 1600 - Report disk usage
- 1690 - Second pass at Mesos containers, with volume sources and network mapping
- 1682 - Ability to update authorized groups separately from full request
- 1695 - Allow the cors bundle to be more configurable
- 1697 - Support uploads to GCS
- 1703 - Better webhook auth timeouts and exception messages
- 1699 - add support for overriding S3 endpoint URL for the downloader
- 1707 - Enable server side encryption params for uploads
- 1717 - Collect cpu usage for tasks that have just started
UI
- 1654 - Ability to specify quick links for requests
- 1636 - New dashboard in Singularity + UI performance improvements
- 1604 - Disk usage ui
- 1687 - Correct copy button on InfoBlocks of Task tab. Also, remove clipboard.js.
- 1704 - Add support for configurable navbar title links.
- 1701 - Permalinks for bounce and scale modals
- 1705 - Capitals search on Singularity requests page
Bug Fixes
- 1609 - Use HostAndPort#getHostText instead of HostAndPort#getHost
- 1658 - Corrected path for executor download fallback
- 1659 - Make sandbox logs/ dir world-readable.
- 1685 - Allow a request in FINISHED state to be redeployed
- 1688 - Also check pending requests on the track task endpoint
- 1693 - Properly send task destroy message to executor
- 1698 - Don't show s3 logs error message as a pop up
- 1708 - Better catch for statusUpdate exceptions
- 1718 - Account for task level overrides in usage collection
- 1719 - Remove unneeded call to unsafeProcessStatusUpdate, fix tasksPerOfferHost check
- 1711 - Take system usage into account when scoring offers
- 1696 - Fix nav bar for mobile view
- 1715 - Use total system cpus, not totalCpus in system calculation
- 1713 - Fix mobile menu responsiveness
- 1726 - Don't operate directly on pending tasks during statusUpdate
- 1721 - Fix cached offer checkin in resourceOffers
Documentation
- 1700 - Fixed task webhook docs
Thanks
- @ssalinas
- @benheng
- @tpetr
- @kwm4385
- @Tamba32
- @baconmania
- @stevegutz
- @stevenschlansker
- @darcatron
Singularity 0.18.2
Changes in 0.18.2
This is a bugfix release
Check out the 0.18.2 milestone to see new features / bugfixes in detail.
Singularity 0.18.1
Changes in 0.18.1
NOTE
It was found that there are issues with static assets in the SingularityService jar for this release, please use 0.18.2 instead
This is a bugfix release
Check out the 0.18.1 milestone to see bugfixes in detail.
Singularity 0.18.0
IMPORTANT
A bug related to missing enum values was found in 0.18.0 please use 0.18.1 instead
Changes in 0.18.0
Check out the 0.18.0 milestone to see new features / bugfixes in detail.
Mesos 1
The release of Singularity 0.18.0 marks an update to mesos 1.x. Singularity will now utilize the mesos http api when connecting to mesos (native libraries no longer need to be present for Singularity to run). Configuration changes for Singularity Service are needed when upgrading so be sure to check out the upgrading to mesos 1 docs.
New Features/Updates
Improvements
- 1631 - Support subdirectories in the s3 uploader
- 1618 - Pass resource overrides to environment.
- 1626 - Expose instance counts for singularity requests
- 1629 - Add shell commands to the client
- 1633 - Make the leniency of OPTIMISTIC tunable.
- 1599 - Add logrotateFrequency field in new deploy form
- 1647 - S3 Folders UI display
Bug Fixes
- 1628 - Preserve instance id order when scaling down
- 1634 - Catch errors in FetchRequestArgHistory
- 1625 - Fixes to latest log file link
- 1630 - Mesos backpressure
- 1621 - look at new scale request before opting to old request
- 1642 - Shade protobuf for SingularityClient/SingularityBase
- 1649 - Don't error when ArtifactManager copies duplicate files
Thanks
- @darcatron
- @baconmania
- @pschoenfelder
- @kwm4385
- @jonathanwgoodwin
Singularity 0.17.0
Changes in 0.17.0
Check out the 0.17.0 milestone to see new features / bugfixes in detail.
Note: Mesos 1.1.x support didn't quite make it into this release, but those changes are coming soon. In the meantime, here are the new features in Singularity 0.17.0.
New Features
- 1592 - Resource Usage UI
- 1576 - Evenly-spread task placement
- 1570 - resource usage endpoint
- 1610 - Highlight new files
Improvements
- 1583 - Track average scheduling delay when accepting tasks.
- 1578 - Allow WORKER <-> SERVICE type change if not load balanced
- 1586 - Refactor tailer instance selection dropdown
- 1615 - Add visibility around getChildren() calls.
- 1597 - Also relocate guava-retrying in SingularityClient
- 1598 - timestampSeconds -> timestamp in statistics object
- 1594 - Leader cache everywhere
- 1590 - package.json tune-up
- 1603 - Reorganize request group view
- 1611 - Add request group filter
- 1608 - Add Nitro as adopter
Bug Fixes
- 1591 - Fix flaky testSchedulerPriority test
- 1593 - More flaky tests
- 1588 - Force Guava cache maintenance before processing cached offers.
- 1587 - Fix key error on requests page
- 1595 - Only write to the leader cache when it's active.
- 1614 - Fix cron length validation when creating new requests.
Thanks
Singularity 0.16.2
Changes in 0.16.2
This is a bug fix release!
Check out the 0.16.2 milestone to see pull requests in detail.
Bug Fixes
- 1605 - Deploy IDs should allow '.'