Media server/FileBackend
From Wikitech
< Media server(Difference between revisions)
RussNelson (Talk | contribs) (→Appends) |
(etherpad paste) |
||
| Line 41: | Line 41: | ||
We (Tim, Bob, and Russ) decided that "Append", which is currently only used by ChunkedUploads, needs to be recast as "concatenate". So whatever kind of chunking happens, it stores a bunch of chunks in the store (probably to the temp zone), and then hands them to concatenate(), which puts them all in the order given into the file given. | We (Tim, Bob, and Russ) decided that "Append", which is currently only used by ChunkedUploads, needs to be recast as "concatenate". So whatever kind of chunking happens, it stores a bunch of chunks in the store (probably to the temp zone), and then hands them to concatenate(), which puts them all in the order given into the file given. | ||
| + | |||
| + | == FileRepo operations == | ||
| + | |||
| + | * publish | ||
| + | ** Move dest -> archive | ||
| + | ** Copy source -> dest | ||
| + | ** Optionally delete source | ||
| + | * store | ||
| + | ** If dest exists, check overwrite mode | ||
| + | ** Copy source -> dest | ||
| + | ** Optionally delete source | ||
| + | * storeTemp | ||
| + | ** Generate name | ||
| + | ** Fail if dest exists | ||
| + | ** Copy source -> dest | ||
| + | * append/concatenate | ||
| + | * freeTemp | ||
| + | ** Delete a file if it exists | ||
| + | * fileExists | ||
| + | * delete | ||
| + | ** Either: | ||
| + | *** Move file to deletion archive, overwrite same | ||
| + | *** Delete file | ||
| + | * getFileProps | ||
| + | * enumFiles | ||
| + | * cleanupDeletedBatch | ||
| + | ** Delete file | ||
| + | |||
| + | == FileBackend basic operations == | ||
| + | |||
| + | src is mwrepo:// or a file:/// url | ||
| + | dest is mwrepo:// | ||
| + | * copy -- fails if the destination exists. | ||
| + | ** ignoreMissingSource | ||
| + | ** overwriteDest | ||
| + | ** overwriteSame - does not actually overwrite, just checks and passes if they are same | ||
| + | ** source | ||
| + | ** dest | ||
| + | * delete -- delete src (ignores zone and destination) | ||
| + | ** source | ||
| + | * move | ||
| + | ** ignoreMissingSource | ||
| + | ** overwriteDest | ||
| + | ** overwriteSame - does not actually overwrite, just checks and passes if they are same | ||
| + | ** source | ||
| + | ** dest | ||
| + | * concatenate | ||
| + | ** source array | ||
| + | ** dest | ||
| + | ** overwriteDest | ||
| + | * fileExists | ||
| + | ** source | ||
| + | * getFileProps | ||
| + | ** source | ||
| + | * enumFiles | ||
| + | * getLocalCopy | ||
| + | ** source | ||
| + | * streamFile | ||
| + | ** source | ||
| + | Copy can accept either a mwrepo:// or a relative local pathname | ||
| + | Delete can accept either a mwrepo:// or a relative local pathname | ||
Latest revision as of 22:04, 16 October 2011
Needed operations are in the following table. In addition, we need three spaces: one which is publicly accessible, and another which is private (for deleted files), and a third which is private and temporary.
| Operation | Existing function name |
|---|---|
| File existence | fileExists |
| File MD5 (for storeBatch) | no |
| Write from filesystem | store |
| Write from virtual URL | publish |
| Read into filesystem | getPath() (doesn't exactly have the right semantics) |
| Open stream | no |
| Get a file's properties | getFileProps (fallback is to read the file into the filesystem and fetch them that way) |
| List files | getThumbnailList (sorta) |
| Delete files | deleteBatch |
| Rename files | LocalFileMoveBatch (although copy && delete is a substitute) |
| Append to a file | append -- may be tricky with some object stores |
| Finish appending to the file | appendFinish |
[edit] Appends
Chunked uploads, and perhaps some other things, require appending to an existing object.
We may need to devise workarounds (such as migrating from one obj to another in a temp space) for those.
- Swift: we know this is possible.
- Azure: we know it's possible from .net; Ben(MS) is checking if can be done from general code
- S3: ???
We (Tim, Bob, and Russ) decided that "Append", which is currently only used by ChunkedUploads, needs to be recast as "concatenate". So whatever kind of chunking happens, it stores a bunch of chunks in the store (probably to the temp zone), and then hands them to concatenate(), which puts them all in the order given into the file given.
[edit] FileRepo operations
- publish
- Move dest -> archive
- Copy source -> dest
- Optionally delete source
- store
- If dest exists, check overwrite mode
- Copy source -> dest
- Optionally delete source
- storeTemp
- Generate name
- Fail if dest exists
- Copy source -> dest
- append/concatenate
- freeTemp
- Delete a file if it exists
- fileExists
- delete
- Either:
- Move file to deletion archive, overwrite same
- Delete file
- Either:
- getFileProps
- enumFiles
- cleanupDeletedBatch
- Delete file
[edit] FileBackend basic operations
src is mwrepo:// or a file:/// url dest is mwrepo://
- copy -- fails if the destination exists.
- ignoreMissingSource
- overwriteDest
- overwriteSame - does not actually overwrite, just checks and passes if they are same
- source
- dest
- delete -- delete src (ignores zone and destination)
- source
- move
- ignoreMissingSource
- overwriteDest
- overwriteSame - does not actually overwrite, just checks and passes if they are same
- source
- dest
- concatenate
- source array
- dest
- overwriteDest
- fileExists
- source
- getFileProps
- source
- enumFiles
- getLocalCopy
- source
- streamFile
- source
Copy can accept either a mwrepo:// or a relative local pathname Delete can accept either a mwrepo:// or a relative local pathname
