OpenStack

Latest revision as of 01:26, 21 January 2013

OpenStack is the architecture we're using for Wikimedia Labs.

[edit] Open tasks

• Write puppet manifests for access control with ssh and sudo
  • Add support to OpenStack manager for sudo access control as described below
• Package gerrit
• Set up gerrit to work as described below
• Write puppet manifests for openstack (mostly done)
• Write puppet config to stop dhcp from spamming the logs on the instances

[edit] Installing

The following is for lucid. They will allow you to set up a multi-node nova configuration.

[edit] On all nova nodes, install the nova PPA

apt-get install python-software-properties
add-apt-repository ppa:nova-core/release
apt-get update

[edit] On the controller, install nova and unrequired dependencies

apt-get install euca2ools nova-api nova-network nova-scheduler nova-ajax-console-proxy glance mysql-server python-mysqldb curl rabbitmq-server dnsmasq euca2ools unzip

[edit] On the compute nodes, install nova and ajaxterm

apt-get install nova-volume nova-compute ajaxterm socat

[edit] On the compute nodes, install sheepdog (optional)

If sheepdog is wanted, you will need to backport libvirt, qemu, and sheepdog and add them to our repo; then on the compute nodes:

apt-get install qemu kvm qemu-kvm qemu-common libjpeg62 libvirt sheepdog corosync

On each additional sheepdog node:

apt-get install sheepdog corosync

On all sheepdog nodes configure corosync according to the sheepdog documentation

[edit] On the LDAP node(s) install opendj

This may be the controller.

apt-get install opendj

Configure opendj according to our LDAP documentation on wikitech. Note: in the section about replacing the global aci, it is important to also disallow access to secretKey and accessKey attributes, otherwise users will be able to see the nova credentials of other users. The syntax for that is:

ds-cfg-global-aci: (targetattr!="userPassword||authPassword||secretKey||access
 Key")(version 3.0; acl "Authenticated read access"; allow (read,search,compar
 e) userdn="ldap:///all";)

You must create an ACI specific to ou=people that gives read access to secretKey and accessKey to the nova wiki user.

[edit] On the MediaWiki node install MediaWiki and dependencies

This may be the controller.

apt-get install php5 php5-mysql php-apc memcached apache2 php5-ldap php5-uuid php5-curl php5-memcache

Configure MediaWiki, installing the LdapAuthentication, OpenStackManager, ParserFunctions, Gadgets, LiquidThreads, CategoryTree, DynamicSidebar, Vector, WikiEditor, SemanticMediaWiki, SemanticForms, SemanticResultFormats, and ExternalData extensions

[edit] On the DNS node(s), install pdns and ldap backend

This can not be on the same system as the network node, unless you do one of the following actions:

• Run pdns on a different port, and add an iptables rule forwarding DNS traffic to pdns
• Bind pdns to the public network IP address, and ensure dnsmasq binds to the host private network IP address

apt-get install pdns-server pdns-backend-ldap

Configure pdns to use ldap-method=strict, and ldap-basedn=ou=hosts,<basedn>; use a proxy agent and password via ldap-binddn and ldap-secret as well

[edit] On the controller and all compute nodes, configure /etc/nova/nova.conf

It should be configured using KVM, FlatDHCP, MySQL, Glance, LDAP, and optionally sheepdog. Here's an example configuration (from a puppet erb template):

--verbose
--daemonize=1
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--sql_connection=mysql://<%= nova_db_user %>:<%= nova_db_pass %>@<%= nova_db_host %>/<%= nova_db_name %>
--image_service=nova.image.glance.GlanceImageService
--s3_host=<%= nova_glance_host %>
--glance_host=<%= nova_glance_host %>
--rabbit_host=<%= nova_rabbit_host %>
--cc_host=<%= nova_cc_host %>
--network_host=<%= nova_network_host %>
--ec2_url=http://<%= nova_api_host %>:8773/services/Cloud
--libvirt_type=kvm
--dhcpbridge=/usr/bin/nova-dhcpbridge
--network_manager=nova.network.manager.FlatDHCPManager
--flat_interface=<% nova_network_flat_interface %>
--public_interface=<% nova_network_public_interface %>
--routing_source_ip=<%= nova_network_public_ip %>
--ajax_console_proxy_url=<%= nova_ajax_proxy_url %>
--volume_driver=nova.volume.driver.SheepdogDriver
--auth_driver=nova.auth.ldapdriver.LdapDriver
--ldap_url=ldap://<% nova_ldap_host %>
--ldap_password=<%= nova_ldap_user_pass %>
--ldap_user_dn=<%= nova_ldap_user_dn %>
--ldap_user_unit=people
--ldap_user_subtree=ou=people,<%= nova_ldap_base_dn %>
--ldap_project_subtree=ou=groups,<%= nova_ldap_base_dn %>
--role_project_subtree=ou=groups,<%= nova_ldap_base_dn %>
--ldap_cloudadmin=cn=cloudadmins,ou=groups,<%= nova_ldap_base_dn %>
--ldap_itsec=cn=itsec,ou=groups,<%= nova_ldap_base_dn %>
--ldap_sysadmin=cn=sysadmins,<%= nova_ldap_base_dn %>
--ldap_netadmin=cn=netadmins,<%= nova_ldap_base_dn %>
--ldap_developer=cn=developers,<%= nova_ldap_base_dn %>

[edit] On the controller, configure /etc/glance/glance.conf

It should be configured using MySQL. Here's an example config (taken from a puppet erb template):

[DEFAULT]
# Show more verbose log output (sets INFO log level output)
verbose = True

# Show debugging output in logs (sets DEBUG log level output)
debug = True

[app:glance-api]
paste.app_factory = glance.server:app_factory

# Which backend store should Glance use by default is not specified
# in a request to add a new image to Glance? Default: 'file'
# Available choices are 'file', 'swift', and 's3'
default_store = file

# Address to bind the API server
bind_host = <%= glance_bind_ip %>

# Port the bind the API server to
bind_port = 9292

# Address to find the registry server
registry_host = <%= glance_bind_ip %>

# Port the registry server is listening on
registry_port = 9191

# ============ Filesystem Store Options ========================

# Directory that the Filesystem backend store
# writes image data to
filesystem_store_datadir=/var/lib/glance/images/

# ============ Swift Store Options =============================

# Address where the Swift authentication service lives
#swift_store_auth_address = 127.0.0.1:8080/v1.0/

# User to authenticate against the Swift authentication service
#swift_store_user = jdoe

# Auth key for the user authenticating against the
# Swift authentication service
#swift_store_key = a86850deb2742ec3cb41518e26aa2d89

# Container within the account that the account should use
# for storing images in Swift
#swift_store_container = glance

# Do we create the container if it does not exist?
#swift_store_create_container_on_put = False

[app:glance-registry]
paste.app_factory = glance.registry.server:app_factory

# Address to bind the registry server
bind_host = <%= glance_bind_ip %>

# Port the bind the registry server to
bind_port = 9191

# SQLAlchemy connection string for the reference implementation
# registry server. Any valid SQLAlchemy connection string is fine.
# See: http://www.sqlalchemy.org/docs/05/reference/sqlalchemy/connections.html#sqlalchemy.create_engine
sql_connection = mysql://<%= glance_db_user %>:<%= glance_db_pass %>@<%= glance_db_host %>/<%= glance_db_name %>

# Period in seconds after which SQLAlchemy should reestablish its connection
# to the database.
#
# MySQL uses a default `wait_timeout` of 8 hours, after which it will drop
# idle connections. This can result in 'MySQL Gone Away' exceptions. If you
# notice this, you can lower this value to ensure that SQLAlchemy reconnects
# before MySQL can drop the connection.
sql_idle_timeout = 3600

[edit] On the controller, create the databases

First, via mysql, create a user for nova, and the nova database. Grant the nova user all permissions on the nova database.

nova-manage db sync

Next, via mysql, create a user for glance, and add the glance database. Grant the glance user all permissions on the glance database.

[edit] On the controller, create the fixed instance network

nova-manage network create 10.4.0.0/24 1 255

Now, you'll need to modify the database to fix the bridge and vlan (it uses br100 and no vlan by default):

mysql -uroot
mysql> use nova;
mysql> update networks set bridge='br103';
mysql> update networks set vlan='103';
mysql> exit

[edit] On the controller, add the nova admin

nova-manage user admin novaadmin admin

Alternatively, add the user via MediaWiki, and add the user to the cloudadmins group. To do this, you'll need to temporarily use Directory Manager to act as the LDAP write user, then after creating and promoting the new user, you can add access controls in LDAP (described below), and change the LDAP write user to be novaadmin, as well as changing the access and secret key.

You should not make this user an Administrator in MediaWiki, since it's better to limit the permissions of the user.

[edit] Give the nova admin access to manage users and groups in LDAP

The following entry will give novadmin full access to the people and groups ou:

dn: ou=people,dc=labs,dc=wikimedia,dc=org                                                                                         
changetype: modify                                                                                                                
replace: aci                                                                                                                      
aci: (targetattr="*") (version 3.0; acl "novaadmin - people"; allow (all)                                                         
 userdn="ldap:///cn=novaadmin,ou=people,dc=labs,dc=wikimedia,dc=org";)                                                            
                                                                                                                                  
dn: ou=groups,dc=labs,dc=wikimedia,dc=org                                                                                         
changetype: modify                                                                                                                
replace: aci                                                                                                                      
aci: (targetattr="*") (version 3.0; acl "novaadmin - people"; allow (all)                                                         
 userdn="ldap:///cn=novaadmin,ou=people,dc=labs,dc=wikimedia,dc=org";)                                                            

dn: ou=hosts,dc=labs,dc=wikimedia,dc=org                                                                                         
changetype: modify                                                                                                                
replace: aci                                                                                                                      
aci: (targetattr="*") (version 3.0; acl "novaadmin - people"; allow (all)                                                         
 userdn="ldap:///cn=novaadmin,ou=people,dc=labs,dc=wikimedia,dc=org";)                                                            

[edit] Create a wiki user for yourself, and promote it using WikiSysop

You'll need to enable '$wgLDAPUseLocal = true;' temporarily to log in as WikiSysop and to create the account. After creating the account, promote it and disable $wgLDAPUseLocal.

[edit] Add a new project via the wiki

Add yourself to the default project, and all project roles.

[edit] Add your account to the global roles via the wiki

The initial user should have all access.

[edit] On the controller, export your users keys and environment

nova-manage project zipfile <projectname> <your-username> <nova.zip-location>

[edit] On the controller, source your environment

unzip nova.zip
source novarc

[edit] On the controller, download a UEC image, and add the image using the glance command

glance add name="ubuntu-<versionnumber>-<versionname>" is_public=true distro="ubuntu" disk_format="qcow2" container_format="ovf" < <image>-disk1.img

[edit] On the gerrit node, install gerrit

useradd -m -d /var/lib/gerrit2 -s /bin/bash gerrit2
su - gerrit2
cd /var/lib/gerrit2
wget http://gerrit.googlecode.com/files/gerrit-2.2.1.war
apt-get install openjdk-6-jre git-core
/usr/bin/java -jar gerrit.war init -d review_site --no-auto-start
# edit /etc/default/gerritcodereview; set GERRIT_SITE, and GERRIT_WAR
exit
cp /var/lib/gerrit2/review_site/bin/gerrit.sh /etc/init.d/gerrit
update-rc.d gerrit defaults

[edit] Project design

[edit] Default project

There will be a default project, which will be a clone of our production cluster. This project is where normal test and development should be done. Root access will generally be limited to ops. Changes can be made to this environment by anyone via the puppet repo.

[edit] As needed projects

All other projects will be created on an as-needed basis. The projects should reflect real-world projects that require new systems infrastructure. Root will be given liberally on instances in these projects. Puppet will not be required for changes on these instances, but will be required for moving the instances into the default project. Instances must be moved into the default project before they are moved into production; as such, all infrastructure must be puppetized before it is moved into production.

[edit] Access control

[edit] Wiki access

User accounts on the wiki (which is also shared credentials for gerrit and nova) can be created by a wiki admin. Requests for access can be made via Ryan Lane, Sara Smollett, Sumana Harihareswara, etc.

To learn how to give out access, see https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ .

[edit] Project and role membership

Initial project and role membership must be assigned by a wiki admin (a member of the ops team, or a trusted delegate). Further project membership can be handled by project members. Further role membership within projects can be handled by role members.

Global role membership can only be assigned by a wiki admin.

[edit] Group membership

Each user has a default group that is shared by all users. Each project is a posix group. When a user is added to a project, they are added to that project's posix group.

Each project will have a sudo role, named <projectname>-sudo. This role will also be a posix group. When a user is added to the role, they will also have the posix group.

[edit] SSH access

SSH access is allowed via SSH keys. Users can add their own keys to their accounts. Access to instances is controlled via access.conf. Users will only be allowed to access instances which belong to projects that the users are a member of.

access.conf will be controlled via puppet. It will use a template to create the access rules via the instanceproject variable, for example:

- : ALL except root (<%= instanceproject %>) : ALL

[edit] Sudo access

Sudo access is granted on a per-project basis via a role. That role is a posix group, so sudo can control access via this group. Puppet will manage sudo via a template. It will grant sudo access to users in the group <projectname>-sudo, for example:

%<%= instanceproject %>-sudo         ALL = (ALL) ALL

[edit] Home directories

Each project will have its own set of home directories, shared via NFS. The home directories for a project will only be shared to instances of that project. The home directories will be created by a script which pulls information from LDAP. The only home directories that will be created will be for users in the project.

The shares file will be managed via a script, that will pull the instance information from LDAP and create a share file; for instance:

/home/<project1> <instance1>,<instance2>(rw)
/home/<project2> <instance3>,<instance4>(rw)

[edit] Project storage

Each project has its own project storage, which is shared via Gluster, and is automounted at /data/project. The /data shares are not ghosted, so do not show up until accessed. Shares are handled this way in anticipation of proper OpenStack Gluster support, where shares will be managed by OpenStack, and volumes can be created at will (to some quota'd point).

The shares are created by a script. The shares are mounted by autofs via an entry in LDAP:

dn: nisMapName=/data,nisMapName=auto.master,dc=wikimedia,dc=org
nisMapName: auto.master
nisMapName: /data
objectClass: top
objectClass: nisObject
nisMapEntry: ldap:nisMapName=auto.data,dc=wikimedia,dc=org
cn: /data

dn: cn=*,nisMapName=auto.data,dc=wikimedia,dc=org
nisMapName: auto.home
objectClass: top
objectClass: nisObject
nisMapEntry: -fstype=glusterfs ${GSERVNAME}:/${PROJECT}-&
cn: *

The autofs variables above are filled in via puppet, where GSERVNAME is the gluster server name used to mount the share, and PROJECT is the instance's project.

[edit] Puppet and Gerrit

Our puppet configuration will be stored in a git repository that is managed by Gerrit. We will have a trunk repo, and a test/dev branch. The trunk repo will be the puppet configuration for the production cluster. The test/dev branch will be the puppet configuration for the default project.

Changes can be made to the puppet configuration the following way:

1. Check out the test/dev branch
2. Create a branch
3. Make changes in the branch
4. Test the changes on the non-default project instance (if this is related to a non-default project)
5. Push the changes
6. Request a merge to the test/dev branch

Then, there will be two approvals required for a merge; one from the community, and one from ops. Ops can bypass the double approval process, if needed. When the merge occurs, the changes can be applied to the default project, and tested. Once the testing is complete, the ops team can merge the changes to the trunk repo, and apply the changes to the production cluster.

Gerrit access will be via ssh keys for git command line, and passwords for the web interface.

[edit] Current Architecture

[edit] Network design

• Tampa
  • Test/dev
    • Private host network
      • Range: 10.4.16.0/24
      • Gateway: 10.4.16.1
      • VLAN: 105
      • Interface: eth0
    • Private guest network
      • Range: 10.4.0.0/24
      • Gateway: 10.4.0.1
      • VLAN: 103
      • Interface: eth1
    • Public Floating IP network
      • Range: 208.80.153.192/26
      • Range: 208.80.153.0/25

• Eqiad
  • • Public Floating IP network
      • Range: 208.80.155.0/24

[edit] Common actions

[edit] Managing floating IP addresses

List current IP addresses

nova-manage floating list

Add a new floating IP address to the pool

nova-manage floating create <last-ip-address+1>

[edit] Managing project quotas

List a project's quota

nova-manage project quota <project-name>

Change a project's floating IP quota

nova-manage project quota <project-name> floating_ips <new-number>

[edit] Troubleshooting

[edit] Rebooting hosts

Please only reboot hosts one at a time, if possible. The hosts are all part of a glusterfs cluster. Rebooting too many at once can cause all of the hosts to have problems.

1. Migrate instances from host
   • If it isn't possible to migrate all instances, ensure you migrate them in order of importance
2. Reboot host
3. Ensure br103 has a proper MAC
   • I've seen a br103 come up with all 0's
4. Ensure dev eth1 and eth1.103 are set to UP
5. Restart nova-compute
6. If you couldn't migrate all instances away, trying rebooting a single instance that is downed
   • Ensure you can ping it
7. Migrate a single instance
   • Ensure you can ping it
8. Migrate the rest of the instances back

[edit] Bringing a dead host back up

1. Ensure br103 has a proper MAC
   • I've seen a br103 come up with all 0's
2. Ensure dev eth1 and eth1.103 are set to UP
3. Restart nova-compute
4. Reboot a single instance that was downed
   • Ensure you can ping it
5. Reboot all instances that were downed

[edit] Rebooting an instance

You can reboot instances via the labsconsole interface, or the euca-reboot-instances command on virt1.

[edit] Fixing a broken live migration

I've run into an issue where live migration failed and didn't properly rollback the migration, either leaving the instance shutdown on all hosts, or migrated without the database properly updated. In either case, it's possible to bring the instance up on the original host, then restart the migration. In the case of an instance being migrated, with the database improperly updated, you'll need to stop the instance on the second host, and start it on the original, then re-migrate.

To do so, copy the instance's nwfilter and qemu definitions from the second host to the original host, add the nwfilter, then create the domain:

virsh nwfilter-define /etc/libvirt/nwfilter/<nwfilter-instance>.xml
virsh create /etc/libvirt/qemu/<nwfilter-qemu>.xml

After doing so, it should be possible to restart the live migration.

You can avoid broken live migrations by only migrating a single instance at a time.

[edit] Fixing an instance that won't reboot

Occasionally an instance my fail to reboot. You can usually solve this by using reboot via nova, but occasionally that fails as well. You can force a reboot by "destroying" the instance then telling nova to reboot the instance. This causes nova to "create" the instance. Of course, "destroy" and "create" really just kill the kvm process and start the process. You should not "delete" or "terminate" the instance.

To force reboot the instance, do the following:

1. Figure out which host the instance is running on
2. Destroy the instance (<instance-id> can be found via virsh list):

   virsh destroy <instance-id>

3. If you see an error like below, then you'll need to restart the libvirt-bin process, then try the destroy

   Timed out during operation: cannot acquire state change lock

4. Tell nova to reboot the instance via "reboot"

[edit] Mounting an instance's disk

Note below that nbd[0-9] means use a number 0-9.

To mount a disk, complete the following steps:

1. Ensure the instance is not running, unless you want to corrupt the disk
2. cd /var/lib/nova/<instance>
3. Mount the disk using qemu-nbd:
   1. If the disk is simply an ext3 image:
      • qemu-nbd -c /dev/nbd[0-9] <disk>
      • mount /dev/nbd[0-9] <mountpoint>
   2. If the disk is a partition inside of an image:
      • qemu-nbd --partition=<partition-number> -c /dev/nbd[0-9] <disk>
      • mount /dev/nbd[0-9] <mountpoint>
   3. If the disk is an LVM volume:
      • qemu-nbd -c /dev/nbd[0-9] <disk>
      • vgscan
      • vgchange -ay
      • mount /dev/<volume-group>/<logical-volume> <mountpoint>

When finished, you should unmount the disk, then disconnect the volume:

1. If the disk is not an LVM volume:
   • umount <mountpoint>
   • qemu-nbd -d /dev/nbd[0-9]
2. If the disk is an LVM volume:
   • umount <mountpoint>
   • vgchange -an <volume-group>
   • qemu-nbd -d /dev/nbd[0-9]

[edit] Running fsck on an instance's disk

First, you'll need the mount the instance's disk. After doing so, you can simply run an fsck against it.

[edit] libvirt won't start, complaining about its certificate

Ensure the certificate file doesn't have whitespace.