Tips for using Hg with (WebFOCUS) reporting? [Was: Betr: Re: Re: Fw: Betr: Re: Many missing revlogs, do we have a problem?]

[Alban Hertroys]

Simon King <simon at simonking.org.uk> wrote on 19/06/2015 12:49:00:


> > In fact, it turns out that ALL the files that hg verify complains 
about
> > are in fact there!
> >
> 
> Somehow, your fncache file was rewritten with all those revlogs
> missing. I was doing some testing and I discovered that old versions
> of mercurial (I tested with 2.5.2) will rewrite the fncache file
> during "hg verify" if the revlogs are missing. This test script shows
> "hg verify" failing even after a missing revlog has been restored:

That suggests that someone here is still using an old version.

I encountered this issue while I was running 3.3 and upgraded my version 
to 3.4.1 and did the same for a colleague (who hadn't been using it yet - 
he's new) in the hope that it was a client-issue and not actual repository 
corruption.

The only other user I could verify is away today. He was on 2.7.something; 
I just upgraded him to 3.4.1 as well.
There's one other user who'se working at home today, I haven't heard back 
from him yet...

> > Is there some way to recreate the fncache in an existing clone?
> 
> Adrian's answer is the official one - clone the repository using "hg
> clone --pull". It is *probably* safe to take the fncache from the
> clone and put it back in the original repo.

That seems to have done the trick. All that remains are some warnings 
saying "hg copy source of ... not in parents of ...", but apparently those 
are harmless.

> From this and your previous emails about file locking, it seems that
> you have a *very unreliable* setup. Mercurial is pretty robust, but it
> does generally assume that the underlying filesystem is sane. If I
> were you I would be looking for ways to fix that.

Well, it's a number of Windows 7 desktops managing a single repo on a CIFS 
mounted file-system that is backed by NTFS on a Windows 2008 R2 server. 
That server is both our development environment and our test/acceptation 
server. On top of that, the server is inside a VMWare instance. I know 
nothing about what hardware that's running on, but apparently the disks 
are on SAS.

Is that (very) unreliable? I would like to say yes, but even I have to 
admit that Windows has become less unreliable over the years (although 
it's still rather primitive compared to other OS'es; terrible 
multi-tasking and terrible USB stack, to name a few).

> For example, it
> seems dangerous that your central repository is also your deployment
> repository. Could you separate those two tasks?

We tried that once, it didn't work out.

The problem is that this repo is inside a directory-tree that contains 
reporting procedures (WebFOCUS), HTML pages and other files, the meta-data 
for the RDBM's we connect to and a significant number of (often quite 
large) de-normalized data-warehouse files. Obviously, the warehouse files 
aren't part of the Hg repository, but they are needed to be able to test 
(and develop) the procedures and other files and should be up-to-date. 
However, we don't properly keep track of where those warehouse-files are; 
most are generated by (more) reporting procedures that are scheduled in 
nightly/weekly or monthly jobs.

To make matters worse, the warehouse-files are mixed in with our reporting 
procedures; directories are organised after departments and tasks rather 
than based on the differences in type of content. We have years of reports 
in our tree, the environment dates back from when the reporting tool was 
not capable of handling sub-directories.

What we tried was to create local development environments on our 
desktops, but we quickly found that it was nigh impossible to have 
relevant (up-to-date) warehouse files. One option was to run the scheduled 
procedures on each desktop as well, but that would multiply the load on 
our database servers - not desirable (big queries!). The other option was 
to find a way to automatically copy the warehouse files from the reporting 
server, but the way they're (not) organised makes that rather difficult.

We should probably reorganise our file hierarchy, but that's a major task. 
We're a small team (4 employees, 2 of which aren't very experienced) and 
we're busy with our main task of creating and maintaining reports. There 
hasn't been much incentive to start this so far; especially if this Hg 
issue turns out to be as temporary as I suspect it is. I'm all for 
reorganising once we have some breathing room, but I think I'll have a 
hard time convincing my superiors.

On the bright side, there's a whole community of Hg users here. There's 
got to be some know-how here about how to organise a WebFOCUS environment 
in such a way that it's easy to create local repo's and have access to 
recent data-warehouse (FOCUS-)files.

Regards,

Alban.


alban.hertroys at apollovredestein.com
T: 




 	Apollo Vredestein B.V. - P.O. Box 27 - 7500 AA  Enschede - The Netherlands - 
Chamber of Commerce number 34223268 - http://www.apollovredestein.com	 

The information contained in this e-mail is intended solely for the use of the 
individual or entity to whom it is addressed and others authorized to receive 
it. You are hereby notified that any disclosure, copying, distribution or 
action in relation to the contents of this information is strictly prohibited. 
If you are not the intended recipient, please delete this message and any 
attachments and advise the sender by return e-mail. The confidentiality of this 
message is not warranted. Apollo Vredestein B.V. rules out any and every 
liability resulting from this or any other electronic transmission.