OpenStack

Revision as of 21:46, 24 April 2011

Open tasks

Write puppet manifests for access control with ssh and sudo
- Add support to OpenStack manager for sudo access control as described below
Package gerrit
Set up gerrit to work as described below
Write puppet manifests for openstack (mostly done)
Write puppet config to stop dhcp from spamming the logs on the instances

Installing

The following is for lucid. They will allow you to set up a multi-node nova configuration.

On all nova nodes, install the nova PPA

apt-get install python-software-properties
add-apt-repository ppa:nova-core/release
apt-get update

On the controller, install nova and unrequired dependencies

apt-get install euca2ools nova-api nova-network nova-scheduler nova-ajax-console-proxy glance mysql-server python-mysqldb curl rabbitmq-server dnsmasq euca2ools unzip

On the compute nodes, install nova and ajaxterm

apt-get install nova-volume nova-compute ajaxterm socat

On the compute nodes, install sheepdog (optional)

If sheepdog is wanted, you will need to backport libvirt, qemu, and sheepdog and add them to our repo; then on the compute nodes:

apt-get install qemu kvm qemu-kvm qemu-common libjpeg62 libvirt sheepdog corosync

On each additional sheepdog node:

apt-get install sheepdog corosync

On all sheepdog nodes configure corosync according to the sheepdog documentation

On the LDAP node(s) install opendj

This may be the controller.

apt-get install opendj

Configure opendj according to our LDAP documentation on wikitech

On the MediaWiki node install MediaWiki and dependencies

This may be the controller.

apt-get install php5 php5-mysql php-apc memcached apache2 php5-ldap php5-uuid php5-curl php5-memcache

Configure MediaWiki, installing the LdapAuthentication, OpenStackManager, ParserFunctions, Gadgets, LiquidThreads, CategoryTree, DynamicSidebar, Vector, WikiEditor, SemanticMediaWiki, SemanticForms, SemanticResultFormats, and ExternalData extensions

On the DNS node(s), install pdns and ldap backend

This can not be on the same system as the network node, unless you do one of the following actions:

Run pdns on a different port, and add an iptables rule forwarding DNS traffic to pdns
Bind pdns to the public network IP address, and ensure dnsmasq binds to the host private network IP address

apt-get install pdns-server pdns-backend-ldap

Configure pdns to use ldap-method=strict, and ldap-basedn=ou=hosts,<basedn>; use a proxy agent and password via ldap-binddn and ldap-secret as well

On the controller and all compute nodes, configure /etc/nova/nova.conf

It should be configured using KVM, FlatDHCP, MySQL, Glance, LDAP, and optionally sheepdog. Here's an example configuration (from a puppet erb template):

--verbose
--daemonize=1
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--sql_connection=mysql://<%= nova_db_user %>:<%= nova_db_pass %>@<%= nova_db_host %>/<%= nova_db_name %>
--image_service=nova.image.glance.GlanceImageService
--s3_host=<%= nova_glance_host %>
--glance_host=<%= nova_glance_host %>
--rabbit_host=<%= nova_rabbit_host %>
--cc_host=<%= nova_cc_host %>
--network_host=<%= nova_network_host %>
--ec2_url=http://<%= nova_api_host %>:8773/services/Cloud
--libvirt_type=kvm
--dhcpbridge=/usr/bin/nova-dhcpbridge
--network_manager=nova.network.manager.FlatDHCPManager
--flat_interface=<% nova_network_flat_interface %>
--public_interface=<% nova_network_public_interface %>
--routing_source_ip=<%= nova_network_public_ip %>
--ajax_console_proxy_url=<%= nova_ajax_proxy_url %>
--volume_driver=nova.volume.driver.SheepdogDriver
--auth_driver=nova.auth.ldapdriver.LdapDriver
--ldap_url=ldap://<% nova_ldap_host %>
--ldap_password=<%= nova_ldap_user_pass %>
--ldap_user_dn=<%= nova_ldap_user_dn %>
--ldap_user_unit=people
--ldap_user_subtree=ou=people,<%= nova_ldap_base_dn %>
--ldap_project_subtree=ou=groups,<%= nova_ldap_base_dn %>
--role_project_subtree=ou=groups,<%= nova_ldap_base_dn %>
--ldap_cloudadmin=cn=cloudadmins,ou=groups,<%= nova_ldap_base_dn %>
--ldap_itsec=cn=itsec,ou=groups,<%= nova_ldap_base_dn %>
--ldap_sysadmin=cn=sysadmins,<%= nova_ldap_base_dn %>
--ldap_netadmin=cn=netadmins,<%= nova_ldap_base_dn %>
--ldap_developer=cn=developers,<%= nova_ldap_base_dn %>

On the controller, configure /etc/glance/glance.conf

It should be configured using MySQL. Here's an example config (taken from a puppet erb template):

[DEFAULT]
# Show more verbose log output (sets INFO log level output)
verbose = True

# Show debugging output in logs (sets DEBUG log level output)
debug = True

[app:glance-api]
paste.app_factory = glance.server:app_factory

# Which backend store should Glance use by default is not specified
# in a request to add a new image to Glance? Default: 'file'
# Available choices are 'file', 'swift', and 's3'
default_store = file

# Address to bind the API server
bind_host = <%= glance_bind_ip %>

# Port the bind the API server to
bind_port = 9292

# Address to find the registry server
registry_host = <%= glance_bind_ip %>

# Port the registry server is listening on
registry_port = 9191

# ============ Filesystem Store Options ========================

# Directory that the Filesystem backend store
# writes image data to
filesystem_store_datadir=/var/lib/glance/images/

# ============ Swift Store Options =============================

# Address where the Swift authentication service lives
#swift_store_auth_address = 127.0.0.1:8080/v1.0/

# User to authenticate against the Swift authentication service
#swift_store_user = jdoe

# Auth key for the user authenticating against the
# Swift authentication service
#swift_store_key = a86850deb2742ec3cb41518e26aa2d89

# Container within the account that the account should use
# for storing images in Swift
#swift_store_container = glance

# Do we create the container if it does not exist?
#swift_store_create_container_on_put = False

[app:glance-registry]
paste.app_factory = glance.registry.server:app_factory

# Address to bind the registry server
bind_host = <%= glance_bind_ip %>

# Port the bind the registry server to
bind_port = 9191

# SQLAlchemy connection string for the reference implementation
# registry server. Any valid SQLAlchemy connection string is fine.
# See: http://www.sqlalchemy.org/docs/05/reference/sqlalchemy/connections.html#sqlalchemy.create_engine
sql_connection = mysql://<%= glance_db_user %>:<%= glance_db_pass %>@<%= glance_db_host %>/<%= glance_db_name %>

# Period in seconds after which SQLAlchemy should reestablish its connection
# to the database.
#
# MySQL uses a default `wait_timeout` of 8 hours, after which it will drop
# idle connections. This can result in 'MySQL Gone Away' exceptions. If you
# notice this, you can lower this value to ensure that SQLAlchemy reconnects
# before MySQL can drop the connection.
sql_idle_timeout = 3600

On the controller, create the databases

First, via mysql, create a user for nova, and the nova database. Grant the nova user all permissions on the nova database.

nova-manage db sync

Next, via mysql, create a user for glance, and add the glance database. Grant the glance user all permissions on the glance database.

On the controller, add the wiki admin

nova-manage user admin wikiadmin admin

Create a wiki user for yourself, and promote it using WikiSysop

You'll need to enable '$wgLDAPUseLocal = true;' temporarily to log in as WikiSysop and to create the account. After creating the account, promote it and disable $wgLDAPUseLocal.

Add a new project via the wiki

Add yourself to the default project, and all project roles.

Add your account to the global roles via the wiki

The initial user should have all access.

On the controller, export your users keys and environment

nova-manage project zipfile <projectname> <your-username> <nova.zip-location>

On the controller, source your environment

unzip nova.zip
source novarc

On the controller, download a UEC image, and add the image using the glance command

/root/glance-uploader.sh -i <image>.img -k <image>-virtual -v "<os version>" -d <distro> -e <kernel version> -a architecture

Project design

Default project

There will be a default project, which will be a clone of our production cluster. This project is where normal test and development should be done. Root access will generally be limited to ops. Changes can be made to this environment by anyone via the puppet repo.

As needed projects

All other projects will be created on an as-needed basis. The projects should reflect real-world projects that require new systems infrastructure. Root will be given liberally on instances in these projects. Puppet will not be required for changes on these instances, but will be required for moving the instances into the default project. Instances must be moved into the default project before they are moved into production; as such, all infrastructure must be puppetized before it is moved into production.

Access control

Wiki access

User accounts on the wiki (which is also shared credentials for gerrit and nova) can be created by a wiki admin. Requests for access can be made via ?

Project and role membership

Initial project and role membership must be assigned by a wiki admin (a member of the ops team, or a trusted delegate). Further project membership can be handled by project members. Further role membership within projects can be handled by role members.

Global role membership can only be assigned by a wiki admin.

Group membership

Each user has a default group that is shared by all users. Each project is a posix group. When a user is added to a project, they are added to that project's posix group.

Each project will have a sudo role, named <projectname>-sudo. This role will also be a posix group. When a user is added to the role, they will also have the posix group.

SSH access

SSH access is allowed via SSH keys. Users can add their own keys to their accounts. Access to instances is controlled via access.conf. Users will only be allowed to access instances which belong to projects that the users are a member of.

access.conf will be controlled via puppet. It will use a template to create the access rules via the instanceproject variable, for example:

- : ALL except root (<%= instanceproject %>) : ALL

Sudo access

Sudo access is granted on a per-project basis via a role. That role is a posix group, so sudo can control access via this group. Puppet will manage sudo via a template. It will grant sudo access to users in the group <projectname>-sudo, for example:

%<%= instanceproject %>-sudo         ALL = (ALL) ALL

Home directories

Each project will have its own set of home directories, shared via NFS. The home directories for a project will only be shared to instances of that project. The home directories will be created by a script which pulls information from LDAP. The only home directories that will be created will be for users in the project.

The shares file will be managed via a script, that will pull the instance information from LDAP and create a share file; for instance:

/home/<project1> <instance1>,<instance2>(rw)
/home/<project2> <instance3>,<instance4>(rw)

Puppet and Gerrit

Our puppet configuration will be stored in a git repository that is managed by Gerrit. We will have a trunk repo, and a test/dev branch. The trunk repo will be the puppet configuration for the production cluster. The test/dev branch will be the puppet configuration for the default project.

Changes can be made to the puppet configuration the following way:

Check out the test/dev branch
Create a branch
Make changes in the branch
Test the changes on the non-default project instance (if this is related to a non-default project)
Push the changes
Request a merge to the test/dev branch

Then, there will be two approvals required for a merge; one from the community, and one from ops. Ops can bypass the double approval process, if needed. When the merge occurs, the changes can be applied to the default project, and tested. Once the testing is complete, the ops team can merge the changes to the trunk repo, and apply the changes to the production cluster.

Gerrit access will be via ssh keys for git command line, and passwords for the web interface.

Current Architecture

Network design

Tampa
- Test/dev
  - Private host network
    - Range: 10.4.16.0/24
    - Gateway: 10.4.16.1
    - VLAN: 105
    - Interface: eth0
  - Private guest network
    - Range: 10.4.0.0/24
    - Gateway: 10.4.0.1
    - VLAN: 103
    - Interface: eth1