Initial commit

.gitignore

@@ -0,0 +1,21 @@
+*.sw[op]
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+.rake_t_cache
+/Gemfile.lock
+/.idea
+/*.iml

Gemfile

@@ -0,0 +1,20 @@
+# -*- encoding: utf-8 -*-
+#
+# Copyright 2014 North Development AB
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source 'https://rubygems.org'
+
+gemspec

README.md

@@ -0,0 +1,120 @@
+# Puppet Forge Server
+
+Private Puppet Forge Server supporting local files and both v1 and v3 API proxies. Heavily inspired by the [Puppet Library](https://github.com/drrb/puppet-library).
+
+Puppet Forge Server provides approximated implementation of both [v1](https://projects.puppetlabs.com/projects/module-site/wiki/Server-api)
+and [v3](https://forgeapi.puppetlabs.com/) APIs, but behavioral deviations from the official implementation might occur.
+
+## Installation
+
+Since the gem is not published yet, please build from source.
+```
+git clone https://github.com/unibet/puppet-forge-server
+cd puppet-forge-server
+bundle install
+bundle exec bin/puppet-forge-server --help
+```
+
+## Getting Started
+
+### Proxy
+
+#### Proxy the official Puppet Forge v3 API
+Just start the server providing the forge URL
+```
+bundle exec bin/puppet-forge-server -x https://forgeapi.puppetlabs.com
+```
+
+Run puppet module install to test it
+```
+puppet module install --module_repository=http://localhost:8080 puppetlabs-stdlib
+```
+
+#### Proxy the official Puppet Forge v1 API and local Pulp puppet repository
+Just start the server providing forge and local pulp URLs
+```
+bundle exec bin/puppet-forge-server -x http://forge.puppetlabs.com -x http://my.local.pulp/pulp_puppet/forge/repository/demo
+```
+
+Run puppet module install to test it. Please note that depending on your internet connection and puppet version it might take a while for the first call to get executed.
+v3 API requires checksuming the files, which means all involved module files will be cached.
+```
+puppet module install --module_repository=http://localhost:8080 puppetlabs-stdlib
+```
+
+### Locally stored modules
+
+Download given modules from the official forge and start the server pointing into directory with module files
+```
+mkdir modules
+wget -P modules/ forge.puppetlabs.com/system/releases/p/puppetlabs/puppetlabs-apache-0.9.0.tar.gz
+wget -P modules/ forge.puppetlabs.com/system/releases/p/puppetlabs/puppetlabs-concat-1.0.0.tar.gz
+wget -P modules/ forge.puppetlabs.com/system/releases/p/puppetlabs/puppetlabs-stdlib-2.4.0.tar.gz
+bundle exec bin/puppet-forge-server -m modules/
+```
+
+Run puppet module install to test it
+```
+puppet module install --module_repository=http://localhost:8080 puppetlabs-stdlib
+```
+
+### All-in
+
+Download given modules from the official forge and start the server pointing into directory with module files and proxy URLs
+```
+mkdir modules
+wget -P modules/ forge.puppetlabs.com/system/releases/p/puppetlabs/puppetlabs-apache-0.9.0.tar.gz
+wget -P modules/ forge.puppetlabs.com/system/releases/p/puppetlabs/puppetlabs-concat-1.0.0.tar.gz
+wget -P modules/ forge.puppetlabs.com/system/releases/p/puppetlabs/puppetlabs-stdlib-2.4.0.tar.gz
+bundle exec bin/puppet-forge-server -m modules/ -x https://forgeapi.puppetlabs.com -x http://my.local.pulp/pulp_puppet/forge/repository/demo
+```
+
+Create an example Puppetfile
+```
+cat > Puppetfile <<EOF
+forge 'http://localhost:8080'
+
+mod 'puppetlabs/apache'
+EOF
+```
+
+Run librarian-puppet with *--no-use-v1-api* option to instruct it to use v3 API
+```
+librarian-puppet install --no-use-v1-api
+```
+## Architecture
+
+Code is structured with MVC in mind to allow easier  maintenance and readability
+
+### API (view)
+
+*API* classes (actually modules only) are used to extend [Sinatra](http://www.sinatrarb.com/) application classes.
+Every module corresponds to official API endpoint and used to present received model data in fasion required by the given API version.
+
+### App (controller)
+
+Every *App* class is a [Sinatra](http://www.sinatrarb.com/) application class and is responsible for mapping API endpoints, querying backends for requested data and providing the results using API (view) modules.
+
+### Models
+
+Puppet module *metadata* json representation is used as a main business *model*.
+
+### Backends
+
+*Backend* classes are providing the means of fetching required data and creating model instances.
+
+
+## TODO
+
+1. Create UTs for core logic
+2. Publish gem at https://rubygems.org/
+3. Implement *source* and *git* backends to match [puppet library](https://github.com/drrb/puppet-library) feature set
+
+## License
+
+[Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0.txt)
+
+## Reference
+
+* [Puppet Library](https://github.com/drrb/puppet-library)
+* [Puppet Anvil](https://github.com/jhaals/puppet-anvil)

bin/puppet-forge-server

@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+#
+# Copyright 2014 North Development AB
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+$:.push File.join(File.dirname(__FILE__), '..', 'lib')
+
+require 'puppet_forge_server'
+
+PuppetForgeServer::Server.new.go(ARGV)

lib/puppet_forge_server.rb

@@ -0,0 +1,63 @@
+# -*- encoding: utf-8 -*-
+#
+# Copyright 2014 North Development AB
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'puppet_forge_server/patches'
+require 'puppet_forge_server/version'
+
+module PuppetForgeServer
+  autoload :Server, 'puppet_forge_server/server'
+  autoload :Errors, 'puppet_forge_server/errors'
+
+  module Api
+    module V1
+      autoload :Modules, 'puppet_forge_server/api/v1/modules'
+      autoload :Releases, 'puppet_forge_server/api/v1/releases'
+    end
+    module V3
+      autoload :Modules, 'puppet_forge_server/api/v3/modules'
+      autoload :Releases, 'puppet_forge_server/api/v3/releases'
+    end
+  end
+
+  module App
+    autoload :Version1, 'puppet_forge_server/app/version1'
+    autoload :Version3, 'puppet_forge_server/app/version3'
+  end
+
+  module Backends
+    autoload :Directory, 'puppet_forge_server/backends/directory'
+    autoload :Proxy, 'puppet_forge_server/backends/proxy'
+    autoload :ProxyV1, 'puppet_forge_server/backends/proxy_v1'
+    autoload :ProxyV3, 'puppet_forge_server/backends/proxy_v3'
+  end
+
+  module Models
+    autoload :Builder, 'puppet_forge_server/models/builder'
+    autoload :Metadata, 'puppet_forge_server/models/metadata'
+  end
+
+  module Utils
+    autoload :Archiver, 'puppet_forge_server/utils/archiver'
+    autoload :OptionParser, 'puppet_forge_server/utils/option_parser'
+    autoload :Url, 'puppet_forge_server/utils/url'
+    autoload :Buffer, 'puppet_forge_server/utils/buffer'
+    autoload :Http, 'puppet_forge_server/utils/http'
+  end
+
+  module Http
+    autoload :HttpClient, 'puppet_forge_server/http/http_client'
+  end
+end

lib/puppet_forge_server/api/v1/modules.rb

@@ -0,0 +1,71 @@
+# -*- encoding: utf-8 -*-
+#
+# Copyright 2014 drrb
+# Copyright 2014 North Development AB
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module PuppetForgeServer::Api::V1
+  module Modules
+
+    def get_modules(metadata)
+      modules = metadata.map do |element|
+        name = element[:metadata].name.sub(/^[^-]+-/, '')
+        full_name = element[:metadata].name.sub('-', '/')
+        {
+            :author => element[:metadata].author,
+            :full_name => full_name,
+            :name => name,
+            :desc => element[:metadata].description,
+            :version => element[:metadata].version,
+            :project_url => element[:metadata].project_page,
+            :releases => [{:version => element[:metadata].version}],
+            :tag_list => [element[:metadata].author, name]
+        }
+      end
+
+      merge_modules(modules)
+    end
+
+    private
+    def merge_modules(modules)
+      grouped_modules = modules.group_by do |result|
+        result[:full_name]
+      end
+
+      grouped_modules.values.map do |value|
+        merge_values(value)
+      end.flatten.uniq
+    end
+
+    def merge_values(value)
+      highest_version, tags, releases = value.inject([nil, [], []]) do |(highest_version, tags, releases), result|
+        [
+            max_version(highest_version, result[:version]),
+            tags + (result[:tag_list] || []),
+            releases + (result[:releases] || [])
+        ]
+      end
+
+      value.first.tap do |result|
+        result[:version] = highest_version
+        result[:tag_list] = tags.uniq
+        result[:releases] = releases.uniq.version_sort_by { |r| r[:version] }.reverse
+      end
+    end
+
+    def max_version(left, right)
+      [Gem::Version.new(left), Gem::Version.new(right)].max.version
+    end
+  end
+end

lib/puppet_forge_server/api/v1/releases.rb

@@ -0,0 +1,31 @@
+# -*- encoding: utf-8 -*-
+#
+# Copyright 2014 North Development AB
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module PuppetForgeServer::Api::V1
+  module Releases
+    def get_releases(metadata)
+      metadata.map do |element|
+        {
+            :file => "/api/v1/files#{element[:path]}",
+            :version => element[:metadata].version,
+            :dependencies => element[:metadata].dependencies.map do |dependency|
+              [dependency['name'], dependency['version_requirement']]
+            end
+        }
+      end.version_sort_by { |r| r[:version] }.reverse
+    end
+  end
+end