dean.edwards.name/weblog/2007/04/packer3/

Comment: #1

I’m sorry but it is still not working with the latest stable prototype.js. When I chek it with spidermonkey, I get the following error:

Desktop/packed.js:1: SyntaxError: missing ; before statement: Desktop/packed.js:1: var Prototype={Version:’1.5.0′,BrowserFeatures:{XPath:!!document.evaluate},ScriptFragment:'(?:<script.*?>)(( |.)*?)(?:<\/script>)’,emptyFunction:function(){},K:function(x){return x}}var Class={create:function(){return function(){this.initialize.apply(

Comment: #2

@Uwe – that is a problem with Prototype. It has a missing semi-colon. The Prototype developers refuse to make Prototype packer-friendly. I don’t know why.

See the packer usage notes for more information.

Comment: #3

Hi Dean,

Here’s a bug, but it’s a corner-case “can I really be bothered to fix this” type of bug – it’s about calling methods on numeric literals:

You already know you can call methods on numbers by bracketing them: alert((12).toFixed(2))

You can save a character by using a single space: alert(12 .toFixed(2))

Packer isn’t aware that the space is important. You could advocate that people use double-dot to achieve the same goal: alert(12..toFixed(2)), or that they simply type the flippin’ string by hand: alert("12.00") (and as if I’ve not wandered off-topic enough, you don’t need the space or double-dot if you’re using a fractional or exponential number: 12e0.toFixed(2) or 12.0.toFixed(2)…)

Comment: #4

@Ash – This is a fixable bug. If I had known about it I would have fixed it in this release. Still, it is not too late. Version 3.0 is in beta at the moment, I can fix it before it goes live. You’re right, it is a bit of an edge-case. No wonder I was unaware of it.

Comment: #5

@Ash – Fixed!

Comment: #6

The Prototype developers refuse to make Prototype packer-friendly. I don’t know why.

I created different Cruncher (PHP) levels for this problem and level 2 removes spaces but it mantains new lines. Had You never think to implement a version, or a flag, that replaces empty spaces or missed semi colons with a “\n” char? I suppose It should resolve prototype (and not only prototype) problems, probably with less compression ratio but more compatiblity.

However, good work, I hope this isn’t an April Fish

Comment: #7

removed the special chars feature (except the ;;; feature which people seem to like)

Does it mean that this option is enabled by default?

Comment: #8

I hope this isn’t an April Fish

Andrea, sometimes your English is priceless.

I don’t intend to add support for missing semi-colons in packer. I thought about it. I even got it working reasonably well. The end result seemed very unpacker like. I like packer the way it is — producing horribly small JavaScript files.

Comment: #9

@Kniaź – yes the option is on by default. In fact, you can no longer it off. Is that OK?

Comment: #10

Andrea, sometimes your English is priceless.

I know it, my English is often (always?) too much horrible, Sorry

Comment: #11

my English is often (always?) too much horrible, Sorry.

It’s a lot better than my Italian. When it comes to foreign languages I am an April Fish.

Comment: #12

Ok, your new function variable substitution makes the $ special character (almost) obsolete, but I still liked the _ special character! Any chance you can add that back in?

Comment: #13

Dean, with regard to semicolons; What if you were to add semicolons back into the origonal source, before packing rather than trying to make packer handle the missing cases.

I believe you could probably use Firefox’s Function.toString to your advantage as i believe if you give it code that doesnt have semicolons, the toStringed version will have had the semicolons inserted into the correct place.

I haven’t looked at the code for the new packer so i dont really know how viable that is sorry.

Comment: #14

just for the sake,here in Italy we (literally) call April fools “April ‘s fish” and we paste paper fish on the back of the others ^_^;;

Comment: #15

@Ash -I had no idea about the 12.. (double dot) or the 12<space> syntax.. I learn something new everyday.

@Dean – when creating the packed versions of Prototype, I found another bug in Safari 2.0.4/Omniweb 5.5.4.

They have a bug in them that causes "High ASCII" encoded text in eval()’s to error out.

For example: eval(‘if(/"/.¡(""));’.replace(/¡/g,’test’));

This bug is not in the nightly builds of WebKit so I suspect it will disappear in the next Safari release.

@Andrea Giammarchi – compressed versions of Prototype (even the 1.5.1rc2 release) can be found at Prototype’s
google group page: http://groups.google.com/group/prototype-core/files

Look for the filename “protopacked_v2d.zip”

Comment: #16

Curse you Safari!

Comment: #17

@Dean – I hear ya. In my case, I just made a shim that detected if the browser could process the little snippet I posted. If they couldn’t I had them load a none high ascii encoded form of Prototype. So a fix may not be needed, just a note that the high ascii encoding effects those browsers.

Comment: #19

Just curious: is packer v. 2.x still going to be available once version 3 is out of beta? I think that version 2 may fit my future needs better, but I’d say that can change depending on what happens with packer v. 3. But anyways, keep up the great work.

Comment: #20

It’s unfortunate that you won’t include the fix for handling “missing” semi-colons. The issue is that Prototype is *valid* Javascript syntax as long as the whitespace/line breaks are maintained so there’s nothing – as far as syntax goes – to “fix.” That’s why the developers balk at adding the semi-colons (granted, they could be nice to the number of compression tools out there by adding them).

IMO, it would be better to ‘fix’ Packer so it handles valid Javascript syntax. Version 4 maybe? Or 3.1?

Comment: #21

@Alex – packer 2 is still available here:

http://dean.edwards.name/packer/2/

Comment: #22

@Dave Myron – I don’t have a problem adding proper semi-colons when I code. I guess it’s a dev preference. If you don’t like using proper semi-colons then js compression will not work well for you. Though, you can always gzip the files with zlib or something.

Comment: #23

@Dave – there are other programs that compress at the level you are describing. You should use JSMin to compress your scripts if you don’t like the strictness of packer.

Comment: #24

In France and Italy, there’s a tradition of sticking paper fish on people’s backs on April Fool’s day. Traditionally, dead fish were used. It’s called poisson d’avril in French and pesce d’aprile in Italian.

Comment: #25

Thanks for this Dean. According to your usage instructions function declarations must be terminated with semi-colons. Is this also required for classes and functions within classes? E.g.

var MyClass
{
function MyFunc()
{
// do something
} [?]
} [?]

Comment: #26

@Chris – That is not valid JavaScript.

Comment: #27

Sorry, I realise I’d written it wrong. I meant something like:

var MyClass
{
MyFunc: function()
{
// do something
} [?]
} [?]

For more details take a look at what I’ll be packing (in a manner of speaking): http://www.stillbreathing.co.uk/projects/performer/performer.html

Comment: #28

@Chris – your code should look like this:

var MyClass = {
  MyFunc: function() {
    // do something
  }
};

Comment: #29

Thanks Dean, you’ve answered my question despite my ridiculous non-code.

Comment: #30

@Dean – have you ever considered making this a server-side component? Something you include that would automatically transform your .js files or a defined block of javascript perhaps.

I ask because having “human readable” code on the server is kinda handy when troubleshooting or bug hunting, but I’d like to pack/obfuscate the code as well.

Comment: #31

@Alex – there are some conversions of packer to languages suitable for this task. There are versions of packer in perl, PHP and .NET. I am also working on a standalone version for Rhino and WSH if that’s what you meant. I’ll release it pretty soon.

Comment: #32

I tried packing the same script with packer v2 and packer v3, it goes from 60 kB -> 20kB in version 2 and version 3 makes a 30kB file. If you want, I can give you the script. Do you have any idea what could be causing this ? Has enyone else noted this ?

Comment: #33

@Ben – packer 3 is still in beta. There is still at least one bug left to fix. Can you send me your source code please?

-dean

Comment: #34

@Dean – not exactly what I meant. I’ve been using the .NET standalone for almost a year now to pack my scripts, but I get errors when obfuscating them (no idea why). If I ported the code to a .NET class and published my source, would that be OK? I’d basically be making the Packer javascript class as a server-side class for authors to use to pack and obfuscate their code at download time instead of at publish time.

Comment: #35

So Dean, no hope for special characters, even the _ one?

Comment: #36

OK. I’ll think about it.

Comment: #37

OK. I’ve thought about it. I’ll put it back. It will be called the “Mills option”. I’m serious.

Edit: I’ve changed my mind again! I might put it back in the next release. You can still use packer version 2 if you like this feature:

http://dean.edwards.name/packer/2/

Comment: #38

Hi Dean. Good to see Packer is still being improved, it’s a very useful tool. There’s one thing missing in this new version that I sorely miss: the ability to browse for a local file to upload. Looks like you can only paste in your code, which is a tad inconvenient. Any chance you might add the feature back in, or am I the only one that uses it?

Comment: #39

@Ambient.Impact – I removed this feature because my server can’t really handle people uploading 1MB javascript files. javascriptcompressor.com hosts a version of packer that will allow this. Currently it is hosting version 2 of packer but will be upgraded when packer 3 is out of beta.

Comment: #40

Wow. That’s a lot of script to upload. Unless you’re exaggerating. Anyhow, I suppose I would make the same decision were my website hosted from my kitchen, what with the limited bandwidth and all.

First I’ve heard of that site, but they seem to be missing the upload feature as well. Oh well, not a huge loss. Thanks for the reply. Keep up the good work.

Comment: #41

@Alex – whose source code did you port? Mine or the .NET source? The .NET conversion was written by Jesse Hansen.

Comment: #42

@Dean – Thanks for considering putting it back anyway.I do use Packer v2. I even updated the .NET version to use an updated generation method for generated names, and I integrated it into my build process. But it’s true, making sure that $ variables don’t conflict is a pain.

Maybe once someone does the .NET conversion for Packer v3, I’ll update my local copy to support the _ special character.

Comment: #43

Hi Dean,

I have also just noted that v3 is not as efficient as v2. My 30k script comes down to 11k using v2, but 19k using v3. Anything I should be looking out for?

Thanks, Chris

Comment: #44

In my results, when you have an inline comment containing a forward slash, some text and another forward slash, it breaks the script because it will include the forward slashes and the contained text in the compressed code.

For example, packing the following jQuery snippet:

$(document).ready(function()
{
	$('#mydiv').doStuff();
	// comment /this will cause errors/
	$('#mydiv').doh();
});

results in:

$(document).ready(function(){$('#mydiv').doStuff();/this will cause errors/$('#mydiv').doh()});

Comment: #45

@everyone – there is a bug in the current beta release use version 2 in the meantime:

http://dean.edwards.name/packer/2/

Comment: #46

@dean – I haven’t ported anyone’s code yet, but I’m asking if it’s alright to port the v3 code to a new .NET class/module (and maybe make a new standalone based on the v3 codebase).

I’ve already begun, however I won’t publish it unless I can get your OK.

Comment: #47

@Alex – Permission granted! Packer 3 is still in beta so it’s probably best to wait for the final release before porting.

Comment: #50

+1 on reenabling the “_” special character – for me that was the feature that differentiated Packer from the rest of the pack.

Still glad you’ve made version 2 available, so I can continue to use that.

Comment: #51

It seems to break under certain circumstances in opera 9.20… I can’t put my finger on it yet, but seeing “undefined undefined” in the (non base62 encoded) result definately isn’t good. It works ok in Firefox.

Comment: #52

@Martijn – that should be fixed now.

Comment: #54

In this code one ; in the for is removed incorrectly:

for(var i = 0; ; i++){
  if(i == 4) break;
  // do something
}

Current result:

for(var i=0;i++){if(i==4)break}

Comment: #55

@jan – I know.

Comment: #56

The tertiary operator does not seem to work when it follow a curly brace. As in

if (blah)
{
}

var path = options.path ? ‘; path=’ + options.path : ”;

Comment: #57

I loved the way packer compresses the scripts and obfuscate them. Here is a question:

If I use it obfuscate my library would (javascript) applications using it would still be valid ?

Comment: #58

hey dean, in your estimation– how trivial would it be to de-obfuscate javascript compressed at the highest level with packer? i.e. does packer offer much defense against code poaching? is it pointless for me to even spend time worrying about it? how do you feel about obsucation as theft protection?

much love for packer!

Comment: #59

Hi Dean:

Great work !!!

I have the same question:

in your estimation– how trivial would it be to de-obfuscate javascript compressed at the highest level with packer?

Thank you

Comment: #60

About that bug with missing semicolon in an empty while loop, can’t you just patch it afterwards with something like this:

js = js.replace(/(while\(.*?\))(})/g, '$1;$2');

Comment: #61

@telepathetic – it’s as trivial as changing ‘eval’ to ‘document.write’

Comment: #62

@wildcard – that would work most of the time but it wouldn’t catch everything. I’m not sure it is worth fixing TBH.

Comment: #63

@telepathetic/thorn – at the highest level (variable replacement) the code would become mostly unreadable as all variables and arguments become single letters.

Comment: #64

Dean, since you brought up variable replacement I thought I’d comment on this post instead of the beta 2 one. When you declare multiple variables at once packer is only obfuscating the first one, leaving the rest unchanged.

eg:

function foo()
{
    var bar = "123", boo, baz = 123;
}

becomes:

function foo(){var a="123",boo,baz=123}

instead of

function foo(){var a="123",b,c=123}

. . .

And on another note, is there any way to add a flag to allow retention of line-ending semicolons?

Comment: #65

@nexus – Yeah I know. It’s too hard to parse the remaining variables so I don’t bother. You must use a separate var statement for each variable that you want to shrink. Also, it doesn’t do function names. I forgot about those.

Comment: #66

And on another note, is there any way to add a flag to allow retention of line-ending semicolons?

OK OK. I get it. More options.

Comment: #67

OK OK. I get it. More options.

Hehe, by no means think that I am (or any one else is for that matter) unappreciative of all the work you’ve done. But yes, more options would be a great addition

I haven’t looked through the code, but I tried to only ask for things I thought would be relatively easy to implement. eg – I assumed that you are using a single regex to remove semicolons so it would be easy to allow a toggle of that functionality.

How are you gathering and obfuscating the variable names? I guess I’ll take a look, mayhaps I’ll come across something I could contribute.

Comment: #68

@nexus – Yeah I know. It’s too hard to parse the remaining variables so I don’t bother. You must use a separate var statement for each variable that you want to shrink.

I had some time to look at the code (ie – I am bored at work) and I made some changes to implement full variable obfuscation regardless of where the variable is declared. I tested it on my Javascript library and it seems to be functioning correctly.

Comment: #69

link?

Comment: #70

I did it on my local machine, so no link. I just changed these lines in _shrinkVariables from:

var VAR_ = /var\s+/g;
var VAR_NAME = /var\s+[\w$]+/g;

. . .

var vars = match(block, VAR_NAME).join(",").replace(VAR_, "");

To:

var VAR_1 = /(\([^\)]*\))/g;
var VAR_2 = /(var\s+)|(=[^,]*)|;/g;
var VAR_LINE = /var\s[^\n\r;]+;/g;

. . .

var vars = match(block, VAR_LINE).join(",").replace(VAR_1,"").replace(VAR_2,"");

You can likely simplify/improve on those expressions; and I didn’t look at how the rest of the code operates so the checks for newline characters and such may be unnecessary.

It should be stable, but aside from a few examples I made to try to break it, the only actual code I tested it on was my Javascript library.

Cheers

Comment: #71

Came across some issues with

for(var xxx in yyy){ . . . }
//and:
var xxx = function(){ . . . }

I made a quick workaround with a negative lookahead:

var VAR_LINE = /var\s+(?![\w$]+\s+in|[\w$]+\s*=\s*function)[^\n\r;]+;|var\s+[\w$]+/g;

It functions correctly in my limited tests; though I’m sure it can be significantly cleaned up (or altered entirely) when a fresh pair of eyes look over it.

And for anyone interested in using this while we wait for its possible introduction into packer, the parentheses in VAR_1 and VAR_2 are unnecessary:

var VAR_1 = /\([^\)]*\)/g;
var VAR_2 = /var\s+|=[^,]*|;/g;

Comment: #72

@nexus – I’ve been working on this myself and came with this:

var BRACKETS =    /\{[^{}]*\}|\[[^\[\]]*\]|\([^\(\)]*\)|~[^~]+~/;
var BRACKETS_g =  global(BRACKETS);
var VARS =        /\bvar\s+[\w$]+[^;]*|\bfunction\s+[\w$]+/g;
var VAR_TIDY =    /\b(var|function)\b|\sin\s+[^;]+/g;
var VAR_EQUAL =   /\s*=[^,;]*/g;
var LIST =        /[^\s,;]+/g;

vars = match(block, VARS).join(";");
while (BRACKETS.test(vars)) {
	vars = vars.replace(BRACKETS_g, "");
}
vars = vars.replace(VAR_TIDY, "").replace(VAR_EQUAL, "");

var ids = match([args, vars], LIST);

It seems to work so far. I’ll include this in the next release of packer.

Comment: #74

Looks good Dean; much cleaner than the stuff I whipped up. I’m looking forward to having this in the official release of packer and glad I could lend a hand (however small) to the project.

Comment: #75

———————–
before
———————–
function test(){
var aaa,bbb;
return aaa+bbb;
}
———————–
after:
———————-
function test(){var a,bbb;return a+bbb}

Maybe, you will add some code to Shrink the ‘bbb’.

Comment: #76

@linb – already improved in the development version:

http://base2.googlecode.com/svn/trunk/src/apps/packer/packer.html

Comment: #77

Is there a reason packer stays disabled? Some days ago it worked for me and I didn’t change anything in my environment (just did a reload of the page).

Comment: #78

@Ralf – I performed site upgrade over the weekend and managed to break packer. It is fixed now.

Comment: #79

Hi Dean, the packer’s still throwing an error I’m afraid:

“ViewCSS.toCamelCase is not a function”

Comment: #80

…and now it’s not, forget my last comment please!

Comment: #81

@Brian – should be working again now.

Comment: #82

Hey Dean, tried Packer, Packer 3, and the one in the trunk. and none of them is now compressing variable names.
I tried: var aaa, bbb;
And all three only removed the whitespace.
Cheers

Comment: #83

Oh.. they are global variables.. I tried a real file and it works.. My apologies

Comment: #84

Decoder for this packer is available in the following URL. In fact this comes next to packer URL in google search for the term “Dean Edwards Packer”

http://yaisb.blogspot.com/2006/10/defeating-dean-edwards-javascript.html

Enjoy.

Comment: #85

Hi

Which version base.2.js should I use for packer? I use packer.js and Words.js with rhino for compress my files. When I use base2.js from http://base2.googlecode.com/svn/trunk/lib/src/base2.js with paker version 3.0 (final) following error happened:

js: “Packer.js”, line 96: uncaught JavaScript runtime exception: ReferenceError: “Array2” is not defined.

Comment: #90

hello first of all thanks for packer … it’s an awesome tool and so far i’m loving every byte of it well … except for 3 of ’em : i’m getting a “‹¯¨” in the middle of the file that throws errors all over the place and i don’t see what i’m doing wrong. i commented out everything … to the point where the file only has this in it :

function R2UQuote(id) {} function CQFailed() {} function CreateNewQuote() {}

can you please point me to what i’m doing wrong here ? thanks a whole bunch !!!

cheers from Canada -Tarik

Comment: #91

me again sorry about that i kinda figured what the problem was. nothing obvious it seems like some kind of corruption on the original file that was not visible in the editor. i deleted the file and recreated a new one with the exact same code and it works. go figure … anyway i’m glad my files are skinny again and you guys rock !

Comment: #92

This would be brilliant as a dreamweaver plugin

Comment: #93

I packed (with variable shrinking) this function

function name1(str) { var name1; };

and got this output

function b(a){var b};

This is a bug, or declaring variable with the same name as the function is a bad practice?

Comment: #96

I noticed a small bug regarding privateVars. I’m using jQuery and ASP.Net and .Net has a habit of mangling element id’s at runtime by concatenating the element hierarchy with underscores. This…

id=”ddRecurFrequency”

…gets turned into this…

id=”ctl00_ContentPlaceHolder1_ddRecurFrequency”

As a workaround, many of my jQuery selectors look like this…

$(“select[id$=’_ddRecurFrequency’]”)

…which is looking for an element who’s id ends with _ddRecurFrequency. When private vars is turned on, it thinks that this is declaring a privatevar with your underscore notation. It looks like that pattern is being a little over selective.

Comment: #100

It rocks!! 5 years of using Dean Edwards Packer and never noticed any bug. Tnx for your gret job, well done!

Comment: #101

Hello there, I doubt you’re reading this page again but it seems like the only ‘logical’ place to post this.

Yes I noticed a bug somehow. It’s basically got to do with the use of a litteral object in a for..in loop. For instance, in recent versions of jQuery:

for(var c in {context:1,url:1})c in b? (…ternary test…)

This will get packed to:

for(var c d{context:1,url:1})c d b? (…)

So the ‘in’ is considered as a variable name. This is certainly due to the regular expression that ‘stops’ after it meets a ‘{‘ sign, when it shouldn’t.

I’m using the PHP version (it works fine once you fix a few things in it — I’ll release the working version in a couple of months, still as MIT). Changing the $VAR_TIDY definition to the following line fixes it:

private $VAR_TIDY = '/\\b(var|function)\\b|\\s(in\\s+[^;{]+|in(?=[);]|$))/';

However, it’s just a hack, not a full-fledged solution. It’s too bad, because Packer is pretty flawless other than that (and the semi-colon issue, to which I also have some fixer code in my version.)

Thanks for your hard work, BTW!

Comment: #102

@Nao, I can’t recreate that problem. Is it only in the PHP version?

Comment: #103

No, it’s in both the PHP version, and version 3.1 as found here: http://base2.googlecode.com/svn/trunk/src/apps/packer/packer.html

Paste the code from jQuery 1.5.2 minified (I suppose it’ll be the same on the unminified version but I tested on that one), and you should see the problem.

It doesn’t happen in version 3.0 (the one hosted on your website). I don’t know why, but I suspect you’d want v3.1 to work the same

Comment: #104

Hello… Just bumping this to ask if you managed to reproduce?

Comment: #105

Yes, I can reproduce it in the version in trunk. I’ve fixed it in my local copy.

Comment: #106

And, err… Any chance you could share your fix with us?Unless you simply re-used the hack I posted earlier…?

Comment: #107

@Nao, can you contact me offline? dean.edwards[AT]gmail.com.

Comment: #108

So err… I sent that email last week from my gmail account but… Nothing?