When I came to Six Fried Rice as a novice developer, I was essentially a completely blank slate in terms of development style and processes. Luckily, Jesse and Geoff are anything but novices at this. They’ve been working with Filemaker long enough to have made all the mistakes I was likely to see, so when it came time to train me on how best to follow the Six Fried Rice methodology they had a pretty broad set of standards and processes ready to go. Those processes have helped me really understand and develop solid, easily understandable and extremely stable systems that if I had been left to my own devices would have taken me years to figure out.
Due to a pretty solid interest that was expressed at this year’s DevCon and a desire to fill in some of those long gaps that tend to pop up on our blog I’m going to be writing several posts that will take you through how we develop at Six Fried Rice from start to finish. There is going to be a combination of more general concepts such as Data Separation and more in depth views on error handling, custom functions and parameter passing. To start it all off though I’m going to start at the absolute beginning. The Data Separation Model.
Data Separation is the concept of isolating the critical business data from the UI and scripts that act upon that data. Here’s an example of how this works out:
Let’s say you have a system that tracks inventory, creates invoices and allows you to enter orders. If you aren’t following the data separation model then you’ve probably got one file containing all of the data tables as well as the scripts and layouts. Any time you need to make a change you have two options. You can either work on the production environment and risk messing up the live data that whoever is using this system relies on or you can work on an offline copy of the file. Both of these methods have some pretty significant disadvantages. Working on the live data of a company isn’t a bad approach for most minor changes and updates, but when you start doing schema changes or drastically altering the scripting that drives the system you run the risk of seriously affecting the business of your client. On the other hand, if you take an offline copy of the files to do major updates you run into the problem of getting those changes back into production. You can’t simply swap files since while you are doing updates your client is making changes to the data in his copy. Testing out the changes offline and then just reproducing them in the client’s file is a huge redundancy. This is where data separation comes in.
Instead of having just one file that contains all the data, scripts and layouts for the entire system let’s divide the data out into its own separate Basefile and stick the layouts and scripts into another file called UI. Now, let’s say you need to do a major system upgrade that’s going to involve changing most of the core scripts in the system and adding a few new layouts. You would just need to get a copy of the UI file, do whatever development and testing you would need against it and when you are ready you can simply swap out the UI files. Since all of the actual data is in its own independant file you don’t need to worry about your client’s data being out of date and you pretty much entirely avoid the problems associated with data migrations. This gives you an exceptional amount of freedom when it comes to trouble shooting and upgrades to your system.
Now, this is not a fool proof system. If the changes you need to make are changes to the actual data structure itself and the relationships that the system needs to function correctly you will still need to do those upgrades to the production system’s Base File. However, there a few development processes and scripting techniques that help to offset this problem and reduce how heavily we rely on the data structures themselves. I will go into how we structure the actual data infrastructure to keep the total number of fields to a minimum in my next post and go over the scripting aspect later on. Basically, we develop around the idea that data gathering and manipulation should be done by passing parameters into a script that is completely context independent, letting the script gather the information or make the change rather than attempting to do so through a relationship or calc field and then having the script return a result that we can make use of if we need it to. We’ll get into this at a deeper level on a later date.
Data separation is one of the key concepts that we integrate into any system that we build. The ability to simply grab a copy of the UI file, make whatever changes we need and test them offline without having to worry about the state of the client’s data makes it possible to avoid quite a few pitfalls. We never have to touch the production environment, data migration is never a problem and development can then be done entirely free of worrying about stepping on user’s toes. The drawbacks include having multiple files that you have to maintain and making changes to the actual fields is a little tedious, but these seem to be small prices to pay for the freedom to manipulate the core system functionality without having to worry about the data.
it is an interesting q, and one i’m considering as i plan a huge db system for my work. i’m definitely going to go separation. But there are two models of doing the above.
i. a server based UI file that accesses server BaseFile
ii. a client based UI file that accesses server BaseFile
I am considering (ii) only because it allows me more freedom to develop from home, and to have people connect from home at speed [only data queries flying about]. Also when i ‘grab a copy’ of the UI file to develop on my computer it is closer to actual normal client experience. There is less read/writing on main disk of UI too?? Scuse any gibberish, I have only looked at FMP9 for 3 months (on and off) now.
U just write a script to say what version you are using, and what version is currently available and then download latest if necessary.
I’m guessing you use (i)?
Or have you guys used both? whaddya recommend?!
blessings,
t
“Since all of the actual data is in it’s own independant file”.
Remember “his/hers/its” and you will never mis-spell “its” again. Or, for consistency, you could adopt the spelling “hi´s, her´s and it´s”.
@rich:
Thanks for the correction 🙂 I remember once when I had misspelled “its”, a coworker corrected me. I said something about how it is impossible to remember which is which since it is completely inconsistent. After all, possessives have apostrophes and so do contractions.
She rocked my world when she replied, “No possessive pronouns have apostrophes.” Of course she was right. I still wonder why none of my English teachers ever pointed out this simple, consistent fact when correcting misuse in my papers. I haven’t been confused about which to use since.
So thanks to an old friend, and to you, for helping us keep things clean here. It is an easy mistake to make, and I missed it while editing.
Geoff
Looking forward to hearing how you make scripts do the job of relationships and calcs! Similar to scripting a web solution for an sql db, no?
Jens
Fascinating. I’ve googled around a bit and found very little information on how to separate the UI/Scripts from the Tables in Filemaker. I’m very much looking forward to part 2!
BTW, one simple question that I could not figure out from my copy of “FMpro8 The missing manual” is whether the relationships go in the file with the tables, or in the file with the UI/Scripts. From experimenting on my own, it appears that they go in the UI/Scripts file.
David:
I’ll address your question about relationships. In general, you need relationships wherever you use them. 🙂 Usually, this is in the UI/Scripts file, because that’s where all the viewing, editing, and scripting of the data occurs. But sometimes you’ll need relationships in the data file as well. For instance, if you define a lookup for one of your fields, since the field definitions live in the data file, you’ll need the relationship the lookup uses in the data file.
In most of our work, the UI/Script file has lots of relationships and the data file has a few.
Geoff
For small projects I prefer all into one self-contained file.
Otman:
There is definitely a threshold below which data separation, while a good practice, simply is not needed. Personally, I like to use a data separated system for just about all of my projects, but it’s more academic in some cases. A few times it’s actually come in amazingly helpful when the system I originally thought was going to be small ends up becoming a fully fleshed out application. Planning for growth even if you don’t see it can be a useful strategy!
Geoff:
Is there a case where a relationship defined in the Data file must be redefined in the UI file too? Does the UI file automatically “know” about relationships already defined in the data file without any extra work?
Mike:
Relationships in the data file do not show up in the UI file in any way. So if a relationship is used in both places, it needs to be defined in both places.
In general relationships in the data file are needed for calculation fields. All other relationships (used for portals, layouts, scripts, etc…) go in the UI file.
Hope that helps.
Geoff
I employ the data separation model wherever possible. It makes life easier, keeps things easy to comprehend and document and keeps the overall complexity of a solution down.
But I don’t think that it really helps for maintaining a system.
Even if you take the best implementation of a system, where there are no calculation fields or relationships used outside of the UI file, it is inevitable that with almost every upgrade (particularly in a larger system) there will be a need to add a new field somewhere. This is true even in PHP/MySQL solutions almost every upgrade will require changes to the MySQL database in some way (it’s just those changes can be scripted).
The problem with this system is that it’s all or nothing: if I even have to change or add a single field outside of the UI file, then all the hard work I’ve done trying to keep things as separate as possible are for nothing.
Note that I’m not knocking the methodology at all; as I said, I use it myself as much as possible. But I’ve yet to find a FM solution where upgrading is as easy as swapping out a file…
I migrated a very large file (over a gigabyte) from a single file configuration to a separated structure, similar to what you described here. Both forms were hosted on the same FileMaker Server 8 Advanced. I experienced a significant degradation in performance. Certain scrips that had been manageable became so slow as to be unusable. I wish I could remember the details of what was slower, but it’s been over a year.
Have you done any comparisons of performance between single file systems and double file systems? Did you encounter speed issues? Have changes been made in FM9 and FM10 that might mitigate such issues?
there is a question: Using Filemaker Server, having a database File,and 2 Interface file all hosted on the Server. there is also a client file that will only open interface file .
the matter of authentication and user acces is a problem for me here.
where should i do all the authentications? how should i pass the current logged on user to diffrent UI or Database File?
what is the solution?
Hi,
I did a Google Search on data separation and hit your comments. I fully support the data separation model, even for smaller solutions and there more reasons than ever for this, because you can easily make two interface files, one for desktop and one for FileMaker Go!
But that’s not the point. I have a question: normally you have N interface files and 1 datafile (or more than 1, but each is unique in that it holds a certain set of data). But how about setting up a test environment? So you have one production database with UI files P1 and P2 pointing to datafile X. And you have a testing database residing on the same server with UI files T1 and T2 pointing to datafile Y.
Users can test whatever they need to in T1 and T2: it’s a test set.
When it’s time to move to production, T1 is renamed to P1 (or P1a) and the external data source for X is redirected to Y.
X and Y are identical, except for their contents (test vs production).
Would this work reliably? I dug my way through the training series, but in my view, this is largely unexplored terrain. Yet with ORACLE or SQL Server, this is common practice!
Thanks,
Hans Erik
Oops,
In my previous post I made a mistake: the external data source ofcourse is redirected from Y (testing) to X (production).
FileMaker 6 and before made a complete mess of data sources! So perhaps that’s a reason most FMP developers I know take it for granted: they don’t dare to look into it, afraid to mess things up…
HE
Hi
I realize this post is from a while back, but I just ran across it. I like the idea of data separation, but have a question about hooking up the interface file to the data file (after you’ve been working on the interface and want to point the new interface file to the production data file). Is there a way to automate changing the file references for each table occurrence, or do you have to do it manually for each TO in the relationship graph?
Or, do you leave your interface file hooked up to the production data file when you are doing development work on it?
Thanks,
Tom
After posting my question, I realize you wouldn’t need to do that sort of thing at all; it looks like you could just keep the same file names, but keep your development data and interface files (with the same names as production versions) in a different directory. Then, when you want to use the new interface, drop the new interface file into the production directory.