Skip to content
This repository has been archived by the owner on Oct 23, 2024. It is now read-only.

Commit

Permalink
Jenkins 4.0.0 Release. (#315)
Browse files Browse the repository at this point in the history
* Make MESOS the default containerizer.

* Make nobody the default user.

* Purge all unncessary plugins except for a minimal set.

* Configureable Jenkins Plugins Installation. (#312)

* User configureable list of Jenkins plugins to be installed at service deployment time.

* Reflect README with the new list of bundled plugins.

* Bump JCasC to 1.35

* Replace config.xml with JCasC variant. Set Mesos to use mesos.leader instead of retreiving it from Zk.

* Add missing JCasC file from previous commit.

* Remove pinned-storage related options.

* Remove pinned storage-hostnames.

* Remove Marathon-LB related configs.

* Re-add volumes support. Split configuration into DC/OS Service, Jenkins-Master and Jenkins-Agent sections respectively.

* Add support for windows and linux jenkins agents. Remove support for additonal plugins (its not stable enough yet)

* Load Mesos plugin from Jenkins CI repo. (#313)

Summary:
The Mesos plugin should be installed from the stable or experimental
repository.

* Move quota enforcement from scripts/bootstrap.py to an external shell-script as a python process can't alter the environment of its calling shell.

* Apply correct DNS name when using frameworks in nested marathon groups.

* Remove Nginx configuration setup from bootstrap.py.

* Export env-var, not the value of the env-var.

* Move creation of ssh-known-scripts out of scripts/bootstrap.sh

* Remove scripts/boostrap.py and with it the Python dependencies. Set default user to nobody.

* Fix missing entries since options restructuring.

* Update base-tech to Jenkins 2.204.2 as the current release has security-vulnerabilities.

* Re-add Marathon-LB settings.

* Readd configureable Jenkins Plugins Installation.

* Mention DCOS_OSS-5906 in the Dockerfile for future cleanups & simplifications.

* Remove unused config/jenkins/config.xml and scripts/bootstrap.py files.

* Remove dependency on libmesos.

* Strict Mode Packaging for USI based Jenkins. (#316)

* WIP: This works, but is messy. Make it easier and more elegant.

* Remove DCOS_SERVICE_ACCOUNT_CREDENTIAL and associated references.

* Re-add service-account, should'nt have been removed before.

* Make private-key free-form as user config can be difficult to predict.

* Use https for mesos leader URL.

* Remove debugging outputs.

* Move jenkins cpu and memory requirements into the jenkins-master config section.

* Add Windows and Linux agent Dockerfiles. (#317)

* Define default entrypoints for Docker images. (#319)

If the Docker in Docker images define the `wrapper.sh` as their default
entrypoint we won't have to use the custom shell in Mesos.

JIRA issues: DCOS_OSS-5937

* Use /login as the health-check endpoint to allow least priveledged users.

* MWT-24 Fixes (#320)

* Update marathon.json.mustache to use strict-mode to set security related settings.

* update jenkins.py to reflect changed options with version 4.0.0

* Change generator-job to be only run on labels of type linux as its the first job run.

* Add support for Marathon group manipulation in sdk_marathon.py

* Update sdk_security to provision secrets for new USI model of extracting secrets for version 4.0.0

* Remove provisioning of Jenkins Slaves, this is done by default when the service is installed. Update secret name to conform with USI. Add support for mesos-agent-labels.

* Mark linux node as a priviledge Dind agent. (#321)

* Mark linux node as a priviledge Dind agent.

* Change Dind image.

* Change DinD image to mesosphere/jenkins-dind:0.8.0

Co-authored-by: Kaiwalya Joshi <[email protected]>

* Add anti-affinity for os:windows Mesos Attribute by default. (#322)

* Set Jenkins linux agent to use mesosphere/jenkins-dind:0.8.0

* Change default agent user to nobody.

* Change default agent user to nobody.

* Define entrypoiht for Windows container.

* Set agent user in load test. (#323)

* Fix error when adding agent_user to options. Make service_user and agent_user configureable.

* Make service-accounts configureable for service_user and agent_user (#324)

* Set permissions for `nobody`
* Make service accounts configureable.

Co-authored-by: Kaiwalya Joshi <[email protected]>

* Set run-delay to 10 to match work-duration.

Co-authored-by: Karsten Jeschkies <[email protected]>
Co-authored-by: Karsten Jeschkies <[email protected]>

* Set cmd as entrypoint for Windows node container. (#325)

Summary:
The command defined in the Jenkins plugin is not compatible with
Powershell. We need to use `cmd /k` instead.

* Use version tagged releases of mesosphere/jenkins-windows-node from now on.

* Bump min DC/OS version to 1.13.

* Bump min DC/OS version to 1.11.

* [D2IQ-67992]  Remove Windows Support. (#326)

* Remove Windows Agents from JCasC.

* Remove jenkins-agent-images.

* Remove Windows Agent Options from Marathon config and Cosmos config.

* Remove os-anti-affinity option required for Windows agents interoperability.

* Use mesos:2.0-beta19 plugin.

* Fixes from the Scale Test Dry-Run (#327)

- Delete any pre-existing secrets before uploading new ones.
- Bump DinD image to `19.03-dind`

* Use mesos:2.0-beta20 plugin.

* Use overlay2 as the storage-driver for DinD image. (#328)

* Use Mesos 2.0 plugin.

* Enforce Jenkins Agent Port Settings.

* Fix Groovy initilization errors.

* Fix startup failures. Bump Jenkins base-tech to 2.204.6 and bump associated plugins.

* Set dind image to 0.9.0

Co-authored-by: Karsten Jeschkies <[email protected]>
Co-authored-by: Karsten Jeschkies <[email protected]>
  • Loading branch information
3 people authored Jul 16, 2020
1 parent c748fe8 commit c6dee30
Show file tree
Hide file tree
Showing 28 changed files with 881 additions and 795 deletions.
29 changes: 14 additions & 15 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM jenkins/jenkins:2.190.1
FROM jenkins/jenkins:2.204.6
WORKDIR /tmp

# Environment variables used throughout this Dockerfile
Expand All @@ -10,15 +10,14 @@ WORKDIR /tmp
ENV JENKINS_FOLDER /usr/share/jenkins

# Build Args
ARG LIBMESOS_DOWNLOAD_URL=https://downloads.mesosphere.io/libmesos-bundle/libmesos-bundle-1.14-beta.tar.gz
ARG BLUEOCEAN_VERSION=1.23.2
ARG BLUEOCEAN_VERSION=1.22.0
ARG JENKINS_STAGING=/usr/share/jenkins/ref/
ARG PROMETHEUS_PLUG_HASH=61ea0cd0bb26d937c8f4df00c7e226c0b51c7b50
ARG STATSD_PLUG_HASH=929d4a6cb3d3ce5f1e03af73075b13687d4879c8
ARG JENKINS_DCOS_HOME=/var/jenkinsdcos_home
ARG user=root
ARG uid=0
ARG gid=0
ARG user=nobody
ARG uid=99
ARG gid=99

ENV JENKINS_HOME $JENKINS_DCOS_HOME
# Default policy according to https://wiki.jenkins.io/display/JENKINS/Configuring+Content+Security+Policy
Expand All @@ -27,11 +26,7 @@ ENV JENKINS_CSP_OPTS="sandbox; default-src 'none'; img-src 'self'; style-src 'se
USER root

# install dependencies
RUN apt-get update && apt-get install -y nginx python zip jq
# libmesos bundle
RUN curl -fsSL "$LIBMESOS_DOWNLOAD_URL" -o libmesos-bundle.tar.gz \
&& tar -C / -xzf libmesos-bundle.tar.gz \
&& rm libmesos-bundle.tar.gz
RUN apt-get update && apt-get install -y nginx python zip jq gettext-base
# update to newer git version
RUN echo "deb http://ftp.debian.org/debian testing main" >> /etc/apt/sources.list \
&& apt-get update && apt-get -t testing install -y git
Expand All @@ -42,17 +37,19 @@ RUN mkdir -p "${JENKINS_HOME}" "${JENKINS_FOLDER}/war"
RUN echo 'networkaddress.cache.ttl=60' >> ${JAVA_HOME}/jre/lib/security/java.security

# bootstrap scripts and needed dir setup
COPY scripts/bootstrap.py /usr/local/jenkins/bin/bootstrap.py
COPY scripts/export-libssl.sh /usr/local/jenkins/bin/export-libssl.sh
COPY scripts/dcos-account.sh /usr/local/jenkins/bin/dcos-account.sh
COPY scripts/dcos-quota.sh /usr/local/jenkins/bin/dcos-quota.sh
COPY scripts/dcos-framework-dns-name.sh /usr/local/jenkins/bin/dcos-framework-dns-name.sh
COPY scripts/dcos-write-known-hosts-file.sh /usr/local/jenkins/bin/dcos-write-known-hosts-file.sh
COPY scripts/run.sh /usr/local/jenkins/bin/run.sh

# nginx setup
RUN mkdir -p /var/log/nginx/jenkins /var/nginx/
COPY conf/nginx/nginx.conf /var/nginx/nginx.conf
COPY conf/nginx/nginx.conf.template /var/nginx/nginx.conf.template

# jenkins setup
COPY conf/jenkins/config.xml "${JENKINS_STAGING}/config.xml"
ENV CASC_JENKINS_CONFIG /usr/local/jenkins/jenkins.yaml
COPY conf/jenkins/configuration.yaml "${CASC_JENKINS_CONFIG}"
COPY conf/jenkins/jenkins.model.JenkinsLocationConfiguration.xml "${JENKINS_STAGING}/jenkins.model.JenkinsLocationConfiguration.xml"
COPY conf/jenkins/nodeMonitors.xml "${JENKINS_STAGING}/nodeMonitors.xml"
COPY scripts/init.groovy.d/mesos-auth.groovy "${JENKINS_STAGING}/init.groovy.d/mesos-auth.groovy"
Expand All @@ -62,6 +59,7 @@ COPY plugins.conf /tmp/
RUN sed -i "s/\${BLUEOCEAN_VERSION}/${BLUEOCEAN_VERSION}/g" /tmp/plugins.conf
RUN /usr/local/bin/install-plugins.sh < /tmp/plugins.conf

# Note: There is a cleaner way of accomplishing the following which is documented in https://jira.d2iq.com/browse/DCOS_OSS-5906
ADD https://infinity-artifacts.s3.amazonaws.com/prometheus-jenkins/prometheus.hpi-${PROMETHEUS_PLUG_HASH} "${JENKINS_STAGING}/plugins/prometheus.hpi"
ADD https://infinity-artifacts.s3.amazonaws.com/statsd-jenkins/metrics-graphite.hpi-${STATSD_PLUG_HASH} "${JENKINS_STAGING}/plugins/metrics-graphite.hpi"

Expand All @@ -74,6 +72,7 @@ RUN chmod -R ugo+rw "$JENKINS_HOME" "${JENKINS_FOLDER}" \
&& chmod -R ugo+rw /var/jenkins_home/ \
&& chmod -R ugo+rw /var/lib/nginx/ /var/nginx/ /var/log/nginx \
&& chmod ugo+rx /usr/local/jenkins/bin/*

USER ${user}

# disable first-run wizard
Expand Down
129 changes: 1 addition & 128 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,134 +21,7 @@ Base packages:
* [Jenkins][jenkins-home] 2.190.1 (LTS)
* [Nginx][nginx-home] 1.10.1

Jenkins plugins:
* ace-editor v1.1
* ansicolor v0.6.2
* ant v1.10
* antisamy-markup-formatter v1.6
* apache-httpcomponents-client-4-api v4.5.10-1.0
* artifactory v3.4.1
* authentication-tokens v1.3
* aws-credentials v1.28
* aws-java-sdk v1.11.636
* azure-commons v1.0.4
* azure-credentials v1.6.1
* azure-vm-agents v1.2.2
* blueocean v1.19.0
* bouncycastle-api v2.17
* branch-api v2.5.4
* build-name-setter v2.0.3
* build-timeout v1.19
* cloudbees-bitbucket-branch-source v2.5.0
* cloudbees-folder v6.9
* cloud-stats v0.25
* command-launcher v1.3
* conditional-buildstep v1.3.6
* config-file-provider v3.6.2
* configuration-as-code v1.31
* copyartifact v1.42.1
* credentials v2.3.0
* credentials-binding v1.20
* cvs v2.14
* display-url-api v2.3.2
* docker-build-publish v1.3.2
* docker-commons v1.15
* docker-workflow v1.19
* durable-task v1.30
* ec2 v1.46.1
* embeddable-build-status v2.0.2
* external-monitor-job v1.7
* favorite v2.3.2
* git v3.12.1
* git-client v2.8.6
* github v1.29.4
* github-api v1.95
* github-branch-source v2.5.8
* github-organization-folder v1.6
* gitlab-plugin v1.5.13
* git-server v1.8
* gradle v1.34
* greenballs v1.15
* handlebars v1.1.1
* handy-uri-templates-2-api v2.1.7-1.0
* htmlpublisher v1.21
* ivy v2.1
* jackson2-api v2.9.10
* javadoc v1.5
* jdk-tool v1.3
* jenkins-design-language v1.19.0
* jira v3.0.10
* jobConfigHistory v2.24
* job-dsl v1.76
* jquery v1.12.4-1
* jquery-detached v1.2.1
* jquery-ui v1.0.2
* jsch v0.1.55.1
* junit v1.28
* ldap v1.20
* lockable-resources v2.5
* mailer v1.29
* mapdb-api v1.0.9.0
* marathon v1.6.0
* matrix-auth v2.4.2
* matrix-project v1.14
* maven-plugin v3.4
* mercurial v2.8
* mesos v1.0.0
* metrics v4.0.2.6
* momentjs v1.1.1
* monitoring v1.79.0
* nant v1.4.3
* node-iterator-api v1.5.0
* pam-auth v1.5.1
* parameterized-trigger v2.35.2
* pipeline-build-step v2.9
* pipeline-github-lib v1.0
* pipeline-graph-analysis v1.10
* pipeline-input-step v2.11
* pipeline-milestone-step v1.3.1
* pipeline-model-api v1.3.9
* pipeline-model-declarative-agent v1.1.1
* pipeline-model-definition v1.3.9
* pipeline-model-extensions v1.3.9
* pipeline-rest-api v2.12
* pipeline-stage-step v2.3
* pipeline-stage-tags-metadata v1.3.9
* pipeline-stage-view v2.12
* plain-credentials v1.5
* prometheus v2.0.6
* pubsub-light v1.13
* rebuild v1.31
* role-strategy v2.14
* run-condition v1.2
* s3 v0.11.2
* saferestart v0.3
* saml v1.1.3
* scm-api v2.6.3
* script-security v1.66
* sse-gateway v1.20
* ssh-agent v1.17
* ssh-credentials v1.17.3
* ssh-slaves v1.30.2
* structs v1.20
* subversion v2.12.2
* timestamper v1.10
* token-macro v2.8
* translation v1.16
* trilead-api v1.0.5
* variant v1.3
* windows-slaves v1.4
* workflow-aggregator v2.6
* workflow-api v2.37
* workflow-basic-steps v2.18
* workflow-cps v2.74
* workflow-cps-global-lib v2.15
* workflow-durable-task-step v2.34
* workflow-job v2.35
* workflow-multibranch v2.21
* workflow-scm-step v2.9
* workflow-step-api v2.20
* workflow-support v3.3
Jenkins plugins: see [plugins.conf](plugins.conf).

## Packaging
Jenkins is available as a package in the [Mesosphere Universe][universe].
Expand Down
78 changes: 0 additions & 78 deletions conf/jenkins/config.xml

This file was deleted.

36 changes: 36 additions & 0 deletions conf/jenkins/configuration.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
jenkins:
agentProtocols:
- "JNLP4-connect"
- "Ping"
numExecutors: 0
clouds:
- mesos:
agentUser: "${JENKINS_AGENT_USER:-nobody}"
frameworkName: "${JENKINS_FRAMEWORK_NAME:-Jenkins Scheduler}"
jenkinsURL: "http://${JENKINS_FRAMEWORK_NAME:-jenkins}.${MARATHON_NAME:-marathon}.mesos:${PORT0:-8080}"
mesosAgentSpecTemplates:
- label: "${JENKINS_LINUX_AGENT_LABEL:-linux}"
agentAttributes: "${JENKINS_LINUX_AGENT_OFFER_SELECTION_ATTRIBUTES:-}"
agentCommandStyle: Linux
containerInfo:
dockerForcePullImage: false
dockerImage: "${JENKINS_LINUX_AGENT_IMAGE:-mesosphere/jenkins-dind:0.9.0}"
dockerPrivilegedMode: true
isDind: true
networking: HOST
type: "DOCKER"
cpus: ${JENKINS_LINUX_AGENT_CPUS:-0.1}
disk: ${JENKINS_LINUX_AGENT_DISK:-0.0}
domainFilterModel: "home"
idleTerminationMinutes: ${JENKINS_LINUX_AGENT_IDLE_TERMINATION_MINUTES:-3}
jnlpArgs: "${JENKINS_LINUX_AGENT_JNLP_ARGS:--noReconnect}"
maxExecutors: ${JENKINS_LINUX_AGENT_MAX_EXECUTORS:-1}
mem: ${JENKINS_LINUX_AGENT_MEM:-512}
minExecutors: ${JENKINS_LINUX_AGENT_MIN_EXECUTORS:-1}
mode: EXCLUSIVE
mesosMasterUrl: "${JENKINS_MESOS_MASTER:-http://leader.mesos:5050}"
role: "${JENKINS_AGENT_ROLE:-*}"
unclassified:
location:
adminAddress: "address not configured yet <nobody@nowhere>"
url: "http://${JENKINS_FRAMEWORK_NAME:-jenkins}.${MARATHON_NAME:-marathon}.mesos:${PORT0:-8080}/"
62 changes: 62 additions & 0 deletions conf/nginx/nginx.conf.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
error_log stderr;
pid /var/nginx/run.pid;

events {
worker_connections 1024;
}

http {
client_max_body_size 1024M;
server {
listen $PORT0 default_server;

access_log /var/log/nginx/jenkins/access.log;
error_log /var/log/nginx/jenkins/error.log;

location ^~ $JENKINS_CONTEXT {
proxy_pass http://127.0.0.1:$PORT1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Connection "";
proxy_max_temp_file_size 0;

# Based on https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy
client_body_buffer_size 128k;

proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}

location ~ ^/(?<url>.*)$ {
rewrite ^/(?<url>.*)$ $JENKINS_CONTEXT/$url break;
proxy_pass http://127.0.0.1:$PORT1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Connection "";
proxy_max_temp_file_size 0;

# Based on https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy
client_body_buffer_size 128k;

proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}

}
}
Loading

0 comments on commit c6dee30

Please sign in to comment.