OpenStack
Open tasks
- Write puppet manifests for access control with ssh and sudo
- Add support to OpenStack manager for sudo access control as described below
- Package gerrit
- Set up gerrit to work as described below
- Write puppet manifests for openstack (mostly done)
- Write puppet config to stop dhcp from spamming the logs on the instances
Installing
The following is for lucid. They will allow you to set up a multi-node nova configuration.
On all nova nodes, install the nova PPA
apt-get install python-software-properties add-apt-repository ppa:nova-core/release apt-get update
On the controller, install nova and unrequired dependencies
apt-get install euca2ools nova-api nova-network nova-scheduler nova-ajax-console-proxy glance mysql-server python-mysqldb curl rabbitmq-server dnsmasq euca2ools unzip
On the compute nodes, install nova and ajaxterm
apt-get install nova-volume nova-compute ajaxterm socatOn the compute nodes, install sheepdog (optional)
If sheepdog is wanted, you will need to backport libvirt, qemu, and sheepdog and add them to our repo; then on the compute nodes:
apt-get install qemu kvm qemu-kvm qemu-common libjpeg62 libvirt sheepdog corosyncOn each additional sheepdog node:
apt-get install sheepdog corosyncOn all sheepdog nodes configure corosync according to the sheepdog documentation
On the LDAP node(s) install opendj
This may be the controller.
apt-get install opendjConfigure opendj according to our LDAP documentation on wikitech. Note: in the section about replacing the global aci, it is important to also disallow access to secretKey and accessKey attributes, otherwise users will be able to see the nova credentials of other users. The syntax for that is:
ds-cfg-global-aci: (targetattr!="userPassword||authPassword||secretKey||access Key")(version 3.0; acl "Authenticated read access"; allow (read,search,compar e) userdn="ldap:///all";)
You must create an ACI specific to ou=people that gives read access to secretKey and accessKey to the nova wiki user.
On the MediaWiki node install MediaWiki and dependencies
This may be the controller.
apt-get install php5 php5-mysql php-apc memcached apache2 php5-ldap php5-uuid php5-curl php5-memcacheConfigure MediaWiki, installing the LdapAuthentication, OpenStackManager, ParserFunctions, Gadgets, LiquidThreads, CategoryTree, DynamicSidebar, Vector, WikiEditor, SemanticMediaWiki, SemanticForms, SemanticResultFormats, and ExternalData extensions
On the DNS node(s), install pdns and ldap backend
This can not be on the same system as the network node, unless you do one of the following actions:
- Run pdns on a different port, and add an iptables rule forwarding DNS traffic to pdns
- Bind pdns to the public network IP address, and ensure dnsmasq binds to the host private network IP address
apt-get install pdns-server pdns-backend-ldapConfigure pdns to use ldap-method=strict, and ldap-basedn=ou=hosts,<basedn>; use a proxy agent and password via ldap-binddn and ldap-secret as well
On the controller and all compute nodes, configure /etc/nova/nova.conf
It should be configured using KVM, FlatDHCP, MySQL, Glance, LDAP, and optionally sheepdog. Here's an example configuration (from a puppet erb template):
--verbose --daemonize=1 --logdir=/var/log/nova --state_path=/var/lib/nova --lock_path=/var/lock/nova --sql_connection=mysql://<%= nova_db_user %>:<%= nova_db_pass %>@<%= nova_db_host %>/<%= nova_db_name %> --image_service=nova.image.glance.GlanceImageService --s3_host=<%= nova_glance_host %> --glance_host=<%= nova_glance_host %> --rabbit_host=<%= nova_rabbit_host %> --cc_host=<%= nova_cc_host %> --network_host=<%= nova_network_host %> --ec2_url=http://<%= nova_api_host %>:8773/services/Cloud --libvirt_type=kvm --dhcpbridge=/usr/bin/nova-dhcpbridge --network_manager=nova.network.manager.FlatDHCPManager --flat_interface=<% nova_network_flat_interface %> --public_interface=<% nova_network_public_interface %> --routing_source_ip=<%= nova_network_public_ip %> --ajax_console_proxy_url=<%= nova_ajax_proxy_url %> --volume_driver=nova.volume.driver.SheepdogDriver --auth_driver=nova.auth.ldapdriver.LdapDriver --ldap_url=ldap://<% nova_ldap_host %> --ldap_password=<%= nova_ldap_user_pass %> --ldap_user_dn=<%= nova_ldap_user_dn %> --ldap_user_unit=people --ldap_user_subtree=ou=people,<%= nova_ldap_base_dn %> --ldap_project_subtree=ou=groups,<%= nova_ldap_base_dn %> --role_project_subtree=ou=groups,<%= nova_ldap_base_dn %> --ldap_cloudadmin=cn=cloudadmins,ou=groups,<%= nova_ldap_base_dn %> --ldap_itsec=cn=itsec,ou=groups,<%= nova_ldap_base_dn %> --ldap_sysadmin=cn=sysadmins,<%= nova_ldap_base_dn %> --ldap_netadmin=cn=netadmins,<%= nova_ldap_base_dn %> --ldap_developer=cn=developers,<%= nova_ldap_base_dn %>
On the controller, configure /etc/glance/glance.conf
It should be configured using MySQL. Here's an example config (taken from a puppet erb template):
[DEFAULT] # Show more verbose log output (sets INFO log level output) verbose = True # Show debugging output in logs (sets DEBUG log level output) debug = True [app:glance-api] paste.app_factory = glance.server:app_factory # Which backend store should Glance use by default is not specified # in a request to add a new image to Glance? Default: 'file' # Available choices are 'file', 'swift', and 's3' default_store = file # Address to bind the API server bind_host = <%= glance_bind_ip %> # Port the bind the API server to bind_port = 9292 # Address to find the registry server registry_host = <%= glance_bind_ip %> # Port the registry server is listening on registry_port = 9191 # ============ Filesystem Store Options ======================== # Directory that the Filesystem backend store # writes image data to filesystem_store_datadir=/var/lib/glance/images/ # ============ Swift Store Options ============================= # Address where the Swift authentication service lives #swift_store_auth_address = 127.0.0.1:8080/v1.0/ # User to authenticate against the Swift authentication service #swift_store_user = jdoe # Auth key for the user authenticating against the # Swift authentication service #swift_store_key = a86850deb2742ec3cb41518e26aa2d89 # Container within the account that the account should use # for storing images in Swift #swift_store_container = glance # Do we create the container if it does not exist? #swift_store_create_container_on_put = False [app:glance-registry] paste.app_factory = glance.registry.server:app_factory # Address to bind the registry server bind_host = <%= glance_bind_ip %> # Port the bind the registry server to bind_port = 9191 # SQLAlchemy connection string for the reference implementation # registry server. Any valid SQLAlchemy connection string is fine. # See: http://www.sqlalchemy.org/docs/05/reference/sqlalchemy/connections.html#sqlalchemy.create_engine sql_connection = mysql://<%= glance_db_user %>:<%= glance_db_pass %>@<%= glance_db_host %>/<%= glance_db_name %> # Period in seconds after which SQLAlchemy should reestablish its connection # to the database. # # MySQL uses a default `wait_timeout` of 8 hours, after which it will drop # idle connections. This can result in 'MySQL Gone Away' exceptions. If you # notice this, you can lower this value to ensure that SQLAlchemy reconnects # before MySQL can drop the connection. sql_idle_timeout = 3600
On the controller, create the databases
First, via mysql, create a user for nova, and the nova database. Grant the nova user all permissions on the nova database.
nova-manage db syncNext, via mysql, create a user for glance, and add the glance database. Grant the glance user all permissions on the glance database.
On the controller, create the fixed instance network
nova-manage network create 10.4.0.0/24 1 255
Now, you'll need to modify the database to fix the bridge and vlan (it uses br100 and no vlan by default):
mysql -uroot mysql> use nova; mysql> update networks set bridge='br103'; mysql> update networks set vlan='103'; mysql> exit
On the controller, add the nova admin
nova-manage user admin novaadmin admin
Alternatively, add the user via MediaWiki, and add the user to the cloudadmins group. To do this, you'll need to temporarily use Directory Manager to act as the LDAP write user, then after creating and promoting the new user, you can add access controls in LDAP (described below), and change the LDAP write user to be novaadmin, as well as changing the access and secret key.
You should not make this user an Administrator in MediaWiki, since it's better to limit the permissions of the user.
Give the nova admin access to manage users and groups in LDAP
The following entry will give novadmin full access to the people and groups ou:
dn: ou=people,dc=labs,dc=wikimedia,dc=org
changetype: modify
replace: aci
aci: (targetattr="*") (version 3.0; acl "novaadmin - people"; allow (all)
userdn="ldap:///cn=novaadmin,ou=people,dc=labs,dc=wikimedia,dc=org";)
dn: ou=groups,dc=labs,dc=wikimedia,dc=org
changetype: modify
replace: aci
aci: (targetattr="*") (version 3.0; acl "novaadmin - people"; allow (all)
userdn="ldap:///cn=novaadmin,ou=people,dc=labs,dc=wikimedia,dc=org";)
dn: ou=hosts,dc=labs,dc=wikimedia,dc=org
changetype: modify
replace: aci
aci: (targetattr="*") (version 3.0; acl "novaadmin - people"; allow (all)
userdn="ldap:///cn=novaadmin,ou=people,dc=labs,dc=wikimedia,dc=org";)
Create a wiki user for yourself, and promote it using WikiSysop
You'll need to enable '$wgLDAPUseLocal = true;' temporarily to log in as WikiSysop and to create the account. After creating the account, promote it and disable $wgLDAPUseLocal.
Add a new project via the wiki
Add yourself to the default project, and all project roles.
Add your account to the global roles via the wiki
The initial user should have all access.
On the controller, export your users keys and environment
nova-manage project zipfile <projectname> <your-username> <nova.zip-location>
On the controller, source your environment
unzip nova.zip source novarc
On the controller, download a UEC image, and add the image using the glance command
/root/glance-uploader.sh -i <image>.img -k <image>-virtual -v "<os version>" -d <distro> -e <kernel version> -a <architecture>
Project design
Default project
There will be a default project, which will be a clone of our production cluster. This project is where normal test and development should be done. Root access will generally be limited to ops. Changes can be made to this environment by anyone via the puppet repo.
As needed projects
All other projects will be created on an as-needed basis. The projects should reflect real-world projects that require new systems infrastructure. Root will be given liberally on instances in these projects. Puppet will not be required for changes on these instances, but will be required for moving the instances into the default project. Instances must be moved into the default project before they are moved into production; as such, all infrastructure must be puppetized before it is moved into production.
Access control
Wiki access
User accounts on the wiki (which is also shared credentials for gerrit and nova) can be created by a wiki admin. Requests for access can be made via ?
Project and role membership
Initial project and role membership must be assigned by a wiki admin (a member of the ops team, or a trusted delegate). Further project membership can be handled by project members. Further role membership within projects can be handled by role members.
Global role membership can only be assigned by a wiki admin.
Group membership
Each user has a default group that is shared by all users. Each project is a posix group. When a user is added to a project, they are added to that project's posix group.
Each project will have a sudo role, named <projectname>-sudo. This role will also be a posix group. When a user is added to the role, they will also have the posix group.
SSH access
SSH access is allowed via SSH keys. Users can add their own keys to their accounts. Access to instances is controlled via access.conf. Users will only be allowed to access instances which belong to projects that the users are a member of.
access.conf will be controlled via puppet. It will use a template to create the access rules via the instanceproject variable, for example:
- : ALL except root (<%= instanceproject %>) : ALL
Sudo access
Sudo access is granted on a per-project basis via a role. That role is a posix group, so sudo can control access via this group. Puppet will manage sudo via a template. It will grant sudo access to users in the group <projectname>-sudo, for example:
%<%= instanceproject %>-sudo ALL = (ALL) ALL
Home directories
Each project will have its own set of home directories, shared via NFS. The home directories for a project will only be shared to instances of that project. The home directories will be created by a script which pulls information from LDAP. The only home directories that will be created will be for users in the project.
The shares file will be managed via a script, that will pull the instance information from LDAP and create a share file; for instance:
/home/<project1> <instance1>,<instance2>(rw) /home/<project2> <instance3>,<instance4>(rw)
Puppet and Gerrit
Our puppet configuration will be stored in a git repository that is managed by Gerrit. We will have a trunk repo, and a test/dev branch. The trunk repo will be the puppet configuration for the production cluster. The test/dev branch will be the puppet configuration for the default project.
Changes can be made to the puppet configuration the following way:
- Check out the test/dev branch
- Create a branch
- Make changes in the branch
- Test the changes on the non-default project instance (if this is related to a non-default project)
- Push the changes
- Request a merge to the test/dev branch
Then, there will be two approvals required for a merge; one from the community, and one from ops. Ops can bypass the double approval process, if needed. When the merge occurs, the changes can be applied to the default project, and tested. Once the testing is complete, the ops team can merge the changes to the trunk repo, and apply the changes to the production cluster.
Gerrit access will be via ssh keys for git command line, and passwords for the web interface.
Current Architecture
Network design
- Tampa
- Test/dev
- Private host network
- Range: 10.4.16.0/24
- Gateway: 10.4.16.1
- VLAN: 105
- Interface: eth0
- Private guest network
- Range: 10.4.0.0/24
- Gateway: 10.4.0.1
- VLAN: 103
- Interface: eth1
- Public Floating IP network
- Range: 208.80.153.192/28
- Private host network
- Test/dev
