Issue 65: Directory Based Allocation.
Project Member Reported by imreckless@gmail.com, Feb 26, 2009
Directory based allocation is to be included in the design.

1. Include absolute paths in the policy data structures.
2. During allocation, do a path look up on the path specified in the
   policy, get its dentry and inode, pass it to ext2_new_inode()
   and ext2_add_nondir().
3. Change the home TID of newly created inode to that of the parent
   directory's home tid.

Feb 28, 2009
Project Member #1 sandeepksinha
Sorry guys for replying lately regarding this.....

Well, Read the DST...
I think, the directory name that you specify must be at the root of the mountpoint 
where you apply the policy.
Eg. <directory>xyz</directory>

You will also have a mechanism to specify if its recursive or non recursive.

Then xyz must be @ /mntpt/xyz

Lets discuss this, and RKS kindly read DST b4 coming.

Mar 2, 2009
Project Member #2 imreckless@gmail.com
Hello Sir,

I have read DST, file placement using namespace.
What i feel is that allocation and relocation is
not recursive or non recursive. Instead relocation 
is recursive and non recursive and its justified.

What it says is that we allocate file of new project 
that is still in testing phase in a cold tier under a specified name space.
When the project is ready and tested it will be relocated to a
production directories. And the selection criteria can be recursive and non recursive.


Mar 2, 2009
Project Member #3 imreckless@gmail.com
Sorry there, the second line is that 
"allocation is not recursive or non recursive"

and not "allocation and relocation is not recursive or non recursive."

Status: In-progress
Mar 3, 2009
Project Member #4 imreckless@gmail.com
Hi all,

Requirements:

1. Directory based allocation.
2. Relocation of directory sub tree
3. Relocation of specific files.

Implementation:

We have no special handling of directories.
Directory will reside in a particular tier, here we have to handle
directories separately. Allocation of directories itself has to be implemented.

Now regular file:
Directory base allocation of files will not require tier to be specified
instead it will inherit tier from parent directory. So in the policy file
the path will be specified instead of tier in this case.

All the module owners kindly identify the changes in you module
with respect to this implementation and see the complexity involved.

Mar 3, 2009
Project Member #5 sandeepksinha
A couple of questions.

What do you mean by Relocation of specific files ?? Its relocation of specific set 
of files. Which also means a directory, because directory holds together a specfic 
set of files.

Again "Allocation of directories itself has to be implemented" makes some other 
sense as compared to our intention.
It should rather be "OHSM's specific changes in the directory allocation needs to be 
done."

Now once again, "So in the policy file
the path will be specified instead of tier in this case.", the path and its home 
tier would be specified. So, when you enable OHSM the directory's home tier ID will 
be set to the specified one in the policy file.
Later for any file which is created inside this directory, it will inherit it from 
the parent directory.

Kindly reframe everything and better come up with doc which discusses all this.
Mar 4, 2009
Project Member #6 imreckless@gmail.com
Relocation of specific set of files is different from relocation of directory 
subtree. 
If i specify relocate all .c files and all .xml files, it is relocation of specific 
files doesnt means a directory.They can be anywhere in FS.
And when i specify relocate directory /mnt/Project from tier 1 to tier 2
it means relocation of directory sub tree.


Another thing,"the path and its home 
tier would be specified. So, when you enable OHSM the directory's home tier ID will 
be set to the specified one in the policy file."

So if you will just set home tid of the specified directory,
initially data blocks of the directory will be somewhere else, 
and when you set the home tid of the directory, new
data blocks will be in the specified tier. This will cause the data blocks of the 
directory to scatter in FS, every time you change the policy.
And reading this directory will be very slow seeking different BGs and if you 
relocate the files in this directory then reading files for relocation will be 
slower.
So directory itself will have to be relocated to the specified tier and
then the allocation of files within this directory.









Mar 4, 2009
Project Member #7 sandeepksinha
NO.

See, the idea is still the same that we had earlier.

========
METHOD 1
========

1. Directory non recursive.
We will ask user to keep this file empty at the time of enabling OHSM.

2. Directory recusrsive.
We will ask user to keep it empty at the time of enabling OHSM.

When you enable OHSM,
---------------------

Take lock on directory inode.
Go and set the home tier id of the directory.
Free the inod lock on directory.

When You create a file
----------------------

Refer to the home tier id of its parent.
If its not zero,
set the home tier id of the new file as teh home tier id of the parent directory.
If its zero,
the creation of new file will follow the file system allocation.

========
METHOD 2
========

If the directory is non-empty at the time of enabling OHSM,

We can just leave the home tier id of directory as 0.
And let it go by the normal allocation.

At the time of relocation, we will relocate everything inside the directory and will 
set the home tier as the new destination, where it is relocated.
Now henceforth, everything that will be allocated inside this directory will be on 
the new destination tier.

I like both the approaches. But which one do you think makes more sense?
I think the code complexity will be same in both the cases, just that in the first 
case we ask the user to have empty directory initially.

Going by the first approach is more flexible as we can ask the user to play a trick.
Suppose the user wants to allocate /mnt/mp3 on tier 1.
And it is not empty.
We can ask him to set a policy for /mnt/mp3files/
And after enabling OHSM, he can just copy the stuffs from /mnt/mp3 to /mnt/mp3files.

This will be better IMHO.


Mar 4, 2009
Project Member #8 imreckless@gmail.com
I think the first approach is simpler and makes more sense,
less complex and more flexible.





Mar 19, 2009
Project Member #9 sandeepksinha
Is this done?
The solution was quite trivial.

I remember RKS doing some tests for this.
Labels: Target-Release1.1 Component-Logic
Apr 24, 2009
#10 rishi.b....@gmail.com
This issue will be dealt with only when coding on OHSM resumes,

putting on hold
Status: On-Hold
Jul 13, 2009
Project Member #11 sandeepksinha
Raising the priority. As far as I remember a initial prototype has been implemented
and tested for sure.

Rohit can I have a update on this? And when we have this completely functional.
From code perspective, looks pretty straight forward to me.


Jul 14, 2009
Project Member #12 imreckless@gmail.com
Directory based allocation has not coded yet,
we were more concentrated on testing relocation that time,
idea is straight forward so it should not take much time.
Changes must be required in policy files, dtd, parser
and admin. And file system related changes will be few.


Jul 15, 2009
Project Member #13 sandeepksinha
I am more concerned about the system code changes. Can you go ahead and provide me
with the code in yours ../users/ directory of svn.

Also, start with just supporting relocation based on directory name. 

Greg, does it make sense to have allocation policies based on directory names? 
IMO, the code complexity is not high, but want more insights on the use cases?


Jul 15, 2009
Project Member #14 imreckless@gmail.com
The suggestion seems nice to me,
any files created under a parent directory whose
home_tid is set will inherit the home_tid from parent.
Jul 15, 2009
Project Member #15 imreckless@gmail.com
This seems to be implemented, i have written a bit of code for it.
For test purpose is set the home_tid in ext2_mkdir() function as 2.
now when i do mkdir on mount point its sets its home tid as 2.
When i create the files inside this directory i simply 
inherit the TID of parent if its set. So this works fine. 
Jul 15, 2009
Project Member #16 sandeepksinha
No the idea here is to simply, 
use the same code as we have today because the code for regular file creation and a
directory is typically same.
So, a single call to ohsm_get_tier, which we already have, should serve our purpose.
Just that we need to make sure that allocpol contains policies for directory as well.
And also, there should be a check that this file's parent directory;s tier is set or not.

Makes sense?
Jul 15, 2009
Project Member #17 sandeepksinha
Do you wish to mark this as work in progress?
Or you want to do it sometimes later?
Jul 15, 2009
Project Member #18 imreckless@gmail.com
Let it stay for a while, as per inheriting home_tid from parent directory, its done,
and for ohsm_get_tier for checking directory based policy it needs overall 
modification. 
Jul 17, 2009
Project Member #19 sandeepksinha
Targeting this to v1.2 
And this issue is no more oh hold.
I think Rohit should start working on this.
Labels: -Target-Release1.1 Target-Release1.2
Jul 18, 2009
Project Member #20 imreckless@gmail.com
Sir , i have started working on it,
as far is inheriting parent directory's its done,

if(EXT2_I(dir)->home_tid)
      TID = EXT2_I(dir)->home_tid;
else
      TID = ohsm_get_tier_for_inode();

this is the code that i added

more over i would say we concentrate more
on recursive relocation, thats a bit of code required in admin.

And for allocation based on 'dir' the check for regular files in ext_creat()
should be removed, that would do policy checking for directory also,

Only policy check for directories is required in admin.

Status: In-progress