Sorting in EnScript – Sorting Arrays and NameListClass / NameValueClass

Every language has its own quirks when it comes to sorting data. In this post, I’ll take an introductory look at some of the most basic methods available for sorting data in EnScript. First, we need a list of some type of data that we want to sort. Our first example is going to use the ulong type, which, in EnScript, is a 64-bit unsigned integer. You might use a ulong if you were storing a list of file sizes, such as those presented by EntryClass::LogicalSize().

We’re going to declare a ulong array type by using the typedef statement as seen in line 3 below. On line 18, we’ll create our array with five values. One of the nicest things about arrays in EnScript is that they have a built-in Sort function, and it takes options from the built-in class ArrayClass. Lines 22, 25, and 28 demonstrate the differences in those options. Using ArrayClass::SORTENABLED performs a default sort of the array in ascending order (smallest to largest). Adding ArrayClass::SORTDESCENDING sorts in descending order (largest to smallest). Finally, adding ArrayClass::SORTNODUPE removes duplicates from the array. This could be useful if you were trying to generate a list of unique values. Take a look through the example below and then check out the output section below it.

ulong Example:

class MainClass {

  typedef ulong[] ulongArray;

  void printArray(ulongArray array) {
    ulong curr;
    forall (ulong u in array) {
      Console.Write(u);
      if (++curr < array.Count()) {
        Console.Write("\t");
      }
    }
    Console.Write("\n");
  }

  void Main() {
    SystemClass::ClearConsole();
    ulongArray array {548, 23, 164, 87, 164};
    //Original order
    printArray(array);
    //Sort ascending
    array.Sort(ArrayClass::SORTENABLED);
    printArray(array);
    //Sort descending
    array.Sort(ArrayClass::SORTENABLED | ArrayClass::SORTDESCENDING);
    printArray(array);
    //Sort ascending, remove duplicates
    array.Sort(ArrayClass::SORTENABLED | ArrayClass::SORTNODUPE);
    printArray(array);
  }
}

Output:

548	23	164	87	164
23	87	164	164	548
548	164	164	87	23
23	87	164	548

Take a look at the output of our code above. Are the results as you expected them? The first line contains the original list in its original order. The second line shows the list after it has been sorted in ascending order, the third after a descending order sort, and the fourth shows the list in ascending order after duplicates have been removed. Pretty straightforward, right? The options are the same for sorting other numerical types of arrays, such as int, char, and even DateClass.

Next let’s take a look at sorting a String array. When you’re dealing with everything in the same case, the options will look much the same as they did above. You’ll note that I threw in a different length string just so you can see how it’s sorted. Scroll down to see the output.

String Example:

class MainClass {

  typedef String[] StringArray;

  void printArray(StringArray array) {
    ulong curr;
    forall (String s in array) {
      Console.Write(s);
      if (++curr < array.Count()) {
        Console.Write("\t");
      }
    }
    Console.Write("\n");
  }

  void Main() {
    SystemClass::ClearConsole();
    StringArray array {"abc", "abb", "aab", "aacd"};
    //Original order
    printArray(array);
    //Sort ascending
    array.Sort(ArrayClass::SORTENABLED);
    printArray(array);
    //Sort descending
    array.Sort(ArrayClass::SORTENABLED | ArrayClass::SORTDESCENDING);
    printArray(array);
  }
}

Output:

abc	abb	aab	aacd
aab	aacd	abb	abc
abc	abb	aacd	aab

You’ll see that it sorts on the first letter, then moves on and sorts on the second letter, and so on. You can see that even though "aacd" is a longer string than "abb" and "abc", it is sorted before them, because the first and second letters "aa" come before "ab" of the latter two strings.

So what happens when we have mixed case strings? You can see on line 19 that all of the array members have the letters "abc" in various mixed case. On line 23 we’re doing a standard ascending sort just like before. On line 26, however, we see a new option: ArrayClass::SORTCASE. This option will turn on case sensitive sorting for strings. You can see its effect in the output below our code.

String Case Sensitive Example:

class MainClass {

  typedef String[] StringArray;

  void printArray(StringArray array) {
    ulong curr;
    forall (String s in array) {
      Console.Write(s);
      if (++curr < array.Count()) {
        Console.Write("\t");
      }
    }
    Console.Write("\n");
  }

  void Main() {
    SystemClass::ClearConsole();
    //Create a new array with mixed case
    StringArray array {"abc", "Abc", "aBC", "abC", "ABc", "AbC"};
    //Original order
    printArray(array);
    //Sort default - case insensitive
    array.Sort(ArrayClass::SORTENABLED);
    printArray(array);
    //Sort case sensitive
    array.Sort(ArrayClass::SORTENABLED | ArrayClass::SORTCASE);
    printArray(array);
  }
}

Output:

abc	Abc	aBC	abC	ABc	AbC
Abc	aBC	abC	ABc	AbC	abc
ABc	AbC	Abc	aBC	abC	abc	

As usual, the first line shows the original order. The second line shows a case insensitive sort – if you converted all of the strings to lowercase, this is the sort you would get. The third line shows our new option – the case sensitive sort. You’ll quickly notice that the first three strings start with a capital "A" and the last three start with a lowercase "a".

The last basic sorting functionality I’ll show you is for NameListClass and NameValueClass. Both of these types inherit from NodeClass, and thus can take advantage of the NodeClass::INSERTSORTED option when inserting a new node. You can see on lines 11 through 16 that we’ve used this option when inserting our NameValueClass objects. This uses an extra line of code for each new object we insert into the list, but it allows us to perform a sorted insert instead of just the order we come upon the values. Both NameListClass and NameValueClass will be sorted on the string value of the Name() property.

NameListClass and NameValueClass Example:

class MainClass {
  void Main() {
    SystemClass::ClearConsole();
    NameValueClass stringList();
    NameValueClass foo1(null, "abc", 0, "foo1");
    NameValueClass bar(null, "abb", 0, "bar");
    NameValueClass foo2(null, "abc", 0, "foo2");
    NameValueClass baz(null, "aab", 0, "baz");
    NameValueClass qux(null, "aac", 0, "qux");
    NameValueClass foo3(null, "abc", 0, "foo3");
    stringList.Insert(foo1, NodeClass::INSERTSORTED);
    stringList.Insert(bar, NodeClass::INSERTSORTED);
    stringList.Insert(foo2, NodeClass::INSERTSORTED);
    stringList.Insert(baz, NodeClass::INSERTSORTED);
    stringList.Insert(qux, NodeClass::INSERTSORTED);
    stringList.Insert(foo3, NodeClass::INSERTSORTED);
    forall (NameValueClass n in stringList) {
      Console.WriteLine(n.Name() + "(" + n.Value() + ")");
    }
  }
}

Output:

aab(baz)
aac(qux)
abb(bar)
abc(foo3)
abc(foo2)
abc(foo1)

You can see that this works in the same manner that our string sort example did. It’s very interesting to note, however, the order that the items with the same value for Name() (foo1, foo2, and foo3) were sorted in. That is – they are sorted in reverse of the order in which they were inserted. The nodes are being inserted just before their value equivalents in the list. I actually find this quite annoying, though as long as we know the behavior we can work around it.

By now you should have a good understanding of the basics of sorting lists in EnScript. In the next post, I’ll show you how to sort arrays of user-defined class objects, and we’ll do some performance tests to see how the built-in array sorting algorithms hold up with large lists.

EnCase 7 Sneak Peek (NYC)

I know a couple of reviews have already been written about the EnCase 7 Sneak Peek as well as a podcast from Forensic 4Cast, but I thought I’d add a few more notes as I caught the New York version on Friday. Consider this primarily a brain dump and I’ll try to add more structure where appropriate. Any wireframes I present below are strictly based on my (Swiss cheese) recollection, and may not be accurate. Also, any use of the phrase “New hotness” should be regarded as having sufficient amounts of sarcasm attached, depending on the article to which it is attached.

New GUI

EnCase 7 is the first major release of Guidance Software’s flagship forensics product in four and a half years (depending on the actual release date) and there are lots of changes, so let’s dive in! The new GUI eliminates the standard File / Edit / View menus in favor of a menu system that gives you quick access to things you use most. Notably, the beloved EnScript pane (which also housed Conditions and Filters) is gone as those have all moved to the top menu bar. I think we were viewing a resolution of around 1280 width and there seemed to be room for all the menus, though, I can’t say that will hold up if more are added.

Old and busted (v6):


New hotness (v7):


The new Home screen which has a web feel to it, showing a list of Recent Cases (with relevant details), New Case, Options, etc. It also has a Logon button, which will presumably be used to customize settings and templates present throughout the new UI for specific users. Not much to say here, but should mean slightly easier access to cases being worked on frequently.

Once a case is opened, the top menu buttons take over: Case, Evidence (Add / Process / Browse), Search, Report. It should be noted that in the demo, cases opened very quickly and there was no apparent re-parsing of past work. This was a promise that Guidance had made in the past for v7 – that investigators wouldn’t have to wait for EnCase to redo all the work they had previously done (resolving search hits, Internet history, etc.). I think they’ve delivered, but the final verdict must be held until beta (or release). Though it wasn’t mentioned in our demo, other blogs have stated that it is using a Btrieve database or a B-tree data structure for the case data. Which it really is… who knows. It was stated in our session that everything gets stored in LEFs (I’m not sure if this was confusion on the presenter’s part and everything is being stored in their new database or in LEFs): expanded compound files; index results; records. I would tend to think it’s getting stored in a structure that allows fast data retrieval, but I won’t speculate on what they’ve decided to use. In any case, the result is that a lot more storage space is required, but things should be more stable and much faster. They’re no longer trying to load everything into memory, which should fix lots of issues with previous versions. I can hear people groaning about the increased storage space now, but I agree with this approach based on their current limitations – storage is cheap (relatively speaking), time is not.

Evidence Processor

Next I should note the new Evidence Processor (replaces Case Processor). This is, of course, an EnScript that executes various modules to perform processing over case evidence. The interface isn’t all that much different that the existing Case Processor, though things are moved around a bit and there are some new options. The focus is on doing more up front and doing it more easily. You can acquire evidence in the Processor, and – Hooray! – drive letters are now associated with physical device numbers. Acquisition can be made using E01 or the new evidence file format, Ex01, which provides native encryption. Interestingly in our session Guidance said the encryption was AES-256, while in other sessions they said AES-128 (“because it was faster than AES-256”), so we’ll probably have to wait to find out. At least this time, the Ex01 will be an open format which Guidance will publish. A bold move from an historically closed company, but welcome nonetheless. Evidence processor will include indexing, which has always been a sore subject since it was introduced in EnCase. No one really used the previous indexing capabilities because they were so unwieldy. This interface makes things a lot easier, but I’ll get more into Search further on. Output of all modules can also be indexed so that you can easily search the results of modules. This might be nice if it’s not too buried. They’re pitching the index as an end all be all link / relationship analysis engine, but I’m not clear on how they see that working in reality.

Evidence Processor includes a list of predefined tasks, and users can setup a template of things they use most often with settings already configured. This is a step forward, albeit a small step. As others have mentioned, processing will be multi-threaded, which is a welcome addition. Some new tasks here are “Thread Email” and “Create Image Thumbnails,” the results of which are stored in LEFs (maybe… see above). Of note, Guidance is working on an integration with Passware, so if you run the Protected File Analysis module in the Processor, and it finds something password protected, it can automatically send the file off to Passware to have it analyzed / cracked. Guidance is claiming it will work with all versions of Passware, but that may change. This will be something with the potential to reduce manual work for investigators. Also, Guidance is supposedly going to make it easier for people to write and plug in custom Processor modules. I sincerely hope Guidance does a better job of documenting this than they have in past releases.

One of the most interesting notes of the evening was that you can run Evidence Processor on multiple examiner boxes and combine the results into one case. This nugget almost went without notice due to the poor presentation skills of our host, but I suppose I happened to perk up at the phrase multiple machines. I would think this should be a huge selling point, but maybe they’re not ready to push it yet. At first the capability will be manual, but they’re working on automating it for a few versions down the line. If they get it right, this will be great! It will almost catch them up to where their competitors were last year, but it’s nice to hear for die-hard EnCase fans. In addition to the Evidence Processor module, they’ll be releasing a dumbed down version of EnCase called EnCase Processor that will only run processing jobs, with no real GUI interaction. For those of you that have used EnCase eDiscovery, this should sound familiar, and I imagine they’re using similar methods.

Evidence Browsing

The old standby Table view of evidence is still around under the heading of “Browse”. This still doesn’t seem to include dates and times from $MFT FileName attribute, and when an audience member asked this question it totally went over the presenter’s head. This view is definitely familiar and long time users will feel right at home here.

Hash Libraries

Hashing isn’t all that glamorous, but there are some new things here as well. There will be a primary and secondary hash library (read: shared and private) that could be useful for labs sharing hash libraries, but also adding to local ones. In version 6 and before, if there were multiple instances of a hash in the library, only the first was shown. In v7, all instances will be shown to give the examiner more information. The examiner will be able to add custom metadata to hash entries (the example given was to associate a case number or conviction to specific hash values). I could definitely put this to use in investigations. MD5 is still supported, as is SHA1. The last new item I noticed is the ability to add a “tag” to hash library entries. I’ll talk more about tags a little later.

Search

Search has perhaps undergone the largest change in the interface. They referred to it as “Unified” search, which isn’t a new term (think Apple’s Spotlight Search). From one location you can search the Index, Tags, and regular Keywords. Searches are now named and the results are saved as such. Think of using multiple labels in Gmail and you’ll understand tags in v7 – you can define up to 63 tags, set colors for them, and then apply them to just about anything in the case. This should allow more flexibility than just bookmarking an item and having to use a folder structure. Hashing, email mounting, Internet history, has all moved to Evidence Processor (I believe).

Old and busted (v6):


New hotness (v7):


Indexing is using the same technology (read: GUI interface) that has been in EnCase eDiscovery for about a year now. They’re boasting integrated “Microsoft word-breaking,” not whitespace delimited, using the most conservative word-breaking. This allows the index to break URLs properly, for example. Searching the index gets you instant hit counts on your search terms (see below). You can also perform an indexed search in specific fields, such as Email To, From, Subject, etc.

Old and busted (v6):


New hotness (v7):


They weren’t prepared to show good old standby binary and GREP keyword searching, but I imagine it hasn’t changed all that much.

Results

Any type of search, including Conditions and Filters, send their results into the… wait for it… Results tab. This is all part of the unified search theme. I’m not terribly sure I like the results navigation just yet. The left side is a list of your “named” searches, conditions, or filters. You can drill into result sets and navigate using the green back and forward buttons that I’m sure you were wondering about in the depiction at the top of the post. I personally found this interface somewhat confusing. I’m hoping it just wasn’t explained very well, otherwise I can see new users having a hard time using the breadcrumbs navigation feature to get around.

New hotness (v7):

Email

Guidance is pitching “E-Mail Review the Way You Need It” with version 7. The default view for Email now is a tree on the left (not super different from how Records look in v6), and the Report view on the right. You can return to the familiar tri-pane view if you’d like. No one seemed clear on how the tree would be sorted by default, but apparently you have to turn the Table pane back on to sort things. They’ve added Conversation Threading and Find Similar functionality which runs across multiple mail files. This was already present in EnCase eDiscovery, and now Forensic users will be able to take advantage of it also. Find Similar isn’t all that advanced, at present it works simply by Subject from what I saw in the demo (and recall from eDiscovery demos at LegalTech), but it’s not bad. They’re supposed to be adding more choices here, like hash value, date ranges, etc. in later versions.

Old and busted (v6):


New hotness (v7):

Reports

There are new Report templates that are supposed to help make reporting easier for users. Headers, footers, colors, font styles, etc. Reports still depend on Bookmarks. There’s a new style editor that looks like a modified version of EnScript that will have average and new users complaining from day 1. Classic Guidance Software – rewrite from scratch when perfectly good markup languages already exist. Oh well, I probably won’t use it, anyway. Some people will love it, some people will hate it. Not sure I see much of a marketplace developing here.

EnScript

Unfortunately the only comments Guidance was prepared to make about EnScript was that there will be a lot more capabilities and that they’ll provide a script to help users translate old scripts to accommodate the new v7 changes. That being said, I think there will be much greater capabilities in v7 EnScript than there were in v6. I can’t wait to see what’s been broken and get to work fixing old scripts.

Neutrino, ProSuite

The Neutrino brand is going away, and all of its features will be integrated directly into EnCase Forensic at no additional cost. Support for non-smartphones also dies with Neutrino, as Guidance realizes that other vendors do this much better. They’re focusing on dealing with operating systems and file systems, so… smartphones. They’re also now supporting iTunes backups and BlackBerry backups (I assume they’re talking about IPD files here). ProSuite components are also being rolled in – EDS, VFS, PDE. Sadly, the old reliable CD/DVD plugin is gone (just kidding, no one ever used that). What this really means is no more separate versions of EnCase aside from Forensic and Enterprise.

Future Additions

Looking to the future: Guidance would like to add OCR capabilities, but wouldn’t mention an integration vendor and wouldn’t say when. Pricing will be available in the next couple weeks, and they hope to release 7.01 a couple months after CEIC.

Summary

Overall, Guidance seems to have made some much needed improvements to speed and stability. Usability increases with the ease of use for index queries. The jury’s still out on the GUI changes (breadcrumb navigation, web look in some screens, unified Results, collapsing EnScripts/Conditions/Filters into a drop-down menu) and multiple machine processing. Anyone who knows me knows my penchant for EnScript, so I’m hoping to see positive changes there. No doubt Jon and I will have lots of fun porting Lightgrep for EnCase to v7.

If you see anything I missed or have questions, please let me know in the comments!