Ok so here we go
First let me extend the peace pipe again - I'm sorry for having been agitated. While on that topic, I'm also sorry it took so long until I had a chance to reply - trust me, I wanted to. I just couldn't get to it, lots of other stuff that unfortunately had higher priorities.
I see you (sivi) linked Schneier's post already, yeah, that was indeed the Schneier I was talking about. But if I might add to your quote, he later added this to his post:
| EDITED TO ADD (5/7): Interesting comments. I think it's worth going through the math. There are two possible ways to do this. The first is to collect, compress, transport, and store. The second is to collect, convert to text, transport, and store. So, what data rates, processing requirements, and storage sizes are we talking about? |
So even Schneier isn't denying that it might be possible, unlike what one might think without reading this update
I'm sure it just wasn't there yet when you posted but still I think it's worth mentioning.
Also Schneier failed to acknowledge the various other sources pointing to such activities, many of which are linked in the comments, among others the AT&T and NSA guys I mentioned before plus a couple of others. Jacob Appelbaum is linked there too, btw, on that topic might I recommend http://assange.rt.com/
, especially the two full length Cypherpunks episodes with Mrs. Appelbaum, Assange, Müller-Maguhn and Zimmermann.
Oh and not to be boastful or anything but I'd like to point out that Schneier apparently failed to take into consideration the voice-to-text aspect before as well but has now and sees the whole thing as less of an impossibility... which btw is surprising, no idea how he could have missed that ("can't store all data...only metadata... oh well, it's worth doing the math").
On that subject IIRC people mentioned different languages, accents etc; that was addressed in the Schneier blog post comments as well, you can easily determine a language and accent first and filter by content later, discard of the most obvious trash etc., then there's data compression that can be done for archival which works beautifully on written (in files) text, etc etc.
I'd really recommend people read the whole blog post, including the comments; some people tried to get the math straight with more or less (seemingly) accurate results; but what it *appears* to come down to is: it is possible, very possible indeed; the cost for storage of all comms are surprisingly low if you take into consideration cloud services and the like, which also makes it obvious that the NSA and similar agencies with similar resources might not really have a hard time financing something like this.
sivi would probably like to stop me right there and say that's not the problem, the aqusition or analysis is; but first off scratch the analysis because that comes later, we're not even that far yet. The whole thread is about the steps before the analysis.
And as for the costs of the aquistion, some interesting devices were already mentioned that are capable of acquiring the required data for surprisingly low prices in relation to the massive surveillance they enable. Again read the Schneier post and especially the comments and the sites linked there, I might pick a few links to repost here later but still, really interesting.
Damn I see "Clive Robinson" (who is a regular poster but I've never felt a need to try and check out who he is) also just mentioned CarrierIQ and similar doing the NSA's job for them there. Oh snap
Lucky for us euroc*nts we seem to not be quite where the US of A are in this respect yet
Oh and btw here's (apparently) the full transcript
of the interview with Clemente on CNN.
Now to the "big post", god help me...
|QUOTE (GrandMaster Smith @ Sunday, May 5 2013, 23:51)|
| now can someone tell me how many democracies have turned into a dictartorship? |
Not actually very many, without external influence or the overthrow of legitimate leaderships by violent non-state actors. In fact, I can't think of a single democracy to dictatorship transition after 1945 which came about because of anything other than the involvement of a foreign power in an internal power struggle.
Just one comment on this, come on, you know precisely what he meant by "turning into a dictatorship". I'm not defending what he said otherwise but getting bogged down in semantics won't help.
|QUOTE (DarrinPA @ Monday, May 6 2013, 01:41)|
| You don't need to. If another terriost attack should happen in America, they'll find out who was involved and then trace back their phone calls, emails, ect, to find who funded the group and to find anymore cells - should there be any. |
You do, though. You've got the entire intelligence process backwards. You don't use intelligence to find out who is involved in a plot after the fact, you use it to break up the plot before it formulates. And if you want to destroy the operating ability of terrorist organisations, militias, criminal cells et cetera, you need to analyse, collate and disseminate the intelligence to law-makers and practitioners rapidly and accurately. Hence why this "scattergun" theory of intelligence is so absurd to anyone trained or experienced in the intelligence world. Why make your already time-dependent job even harder by collecting vast quantities of completely useless information when there are specialist analysts whose sole role it is to tell the people responsible for intelligence collection what to collect?
Well, I'm not sure about the example Darrin gave but there certainly are cases in intelligence where the process works backwards as you refer to it. Like Boston appears to become now. Or the case where the sheik was killed in his hotel by what, 15 or so Mossad agents dressed as tourists?
Of course maybe the definition of "intelligence" in this context just escapes me as a non-native speaker but from what I gather it's nothing but another word for gathering (and processing, analyzing etc.) information in an effort to find out something that can help against an adversary of some sort. It doesn't necessarily mean that the adversary can't already have acted, right? If it does my apologies.
But really I think me and Darrin weren't debating the precise order of events in usual intelligence cases but more that the capabilities for such new kinds of massive surveillance might exist, or actually that we find it probable that they do.
| One, it's not an article. It's a comment piece. |
| It's a comment piece that's actually much less clear cut than you imply:|
|QUOTE (CNN @ 1st May 2013)|
| CLEMENTE: "No, there is a way. We certainly have ways in national security investigations to find out exactly what was said in that conversation. It's not necessarily something that the FBI is going to want to present in court, but it may help lead the investigation and/or lead to questioning of her. We certainly can find that out. |
Which, followed by his second comment, isn't actually tantamount to "we record absolutely everything, ever". That's certainly how our good friend Glenn Greenwald at The Guardian interprets it, but that's not actually what's being said in the interview. There's certainly the capability to recover an awful lot of valuable data from phone calls, but nowhere does anyone with any experiences of SIGINT categorically say that every phone call, electronic communication and web browsing session recorded.
To be entirely fair, let's get the entire quote from the guardian in here, important parts bolded to ruthlessly further my arguments:
| BURNETT: Tim, is there any way, obviously, there is a voice mail they can try to get the phone companies to give that up at this point. It's not a voice mail. It's just a conversation. There's no way they actually can find out what happened, right, unless she tells them?|
CLEMENTE: "No, there is a way. We certainly have ways in national security investigations to find out exactly what was said in that conversation. It's not necessarily something that the FBI is going to want to present in court, but it may help lead the investigation and/or lead to questioning of her. We certainly can find that out.
BURNETT: "So they can actually get that? People are saying, look, that is incredible.
CLEMENTE: "No, welcome to America. All of that stuff is being captured as we speak whether we know it or like it or not."
So to pick out the most important points:
- "a conversation", aka voice;
- "find out what happened", which obviously means the content of said conversation
- "have ways to find out exactly what was said
in that conversation" - any more questions? He is NOT talking about metadata. He is not talking about trying to reconstruct something to get a rudimentary understanding of the content. He is talking about finding out exactly what was said
, that means word for word, that means it has to be saved somewhere, whether temporarily or forever is anyone's guess. The same thing goes for the format, OPUS, ASCII or whatever.
- "No, welcome to America" clearly implies that this is standard procedure, not something that is only targeted at very specific people
- "All of that stuff is being captured as we speak whether we know it or like it or not" is also not really open for interpretation now is it?!
So tell me then how you would reasonably interpret what he said? I mean if anything I'd have expected you to tell me the guy wasn't credible for whatever reasons, despite even the US govt not having said anything about the matter as far as I'm aware, again despite the waves the topic seems to start creating around certain places.
But attacking Greenwald and saying his interpretation was somehow weird... I just don't see where. I'm guilty of attacking you as well so I can hardly blame you but I think generally we can now agree to keep it more civil, the same IMHO goes for the people we're talking about.
| A word about Mr Greenwald, if you will. He's no an intelligence expert. He's not even that versed on foreign policy. He's a constitutional lawyer with a string of rather provocative books to his name, bearing titles like With Liberty and Justice for Some: How the Law Is Used to Destroy Equality and Protect the Powerful. A noted columnist whose works are certainly interesting, yes, but are his opinions- this being present in the "Comment Is Free" section of the website- the gospel truth? Lets look a little further into it in the following post. |
Hm, afaik you didn't get back to him yet, not that I need or want you to. Tbh I'm not interested in whatever your opinion is on Greenwald because his opinion did not influence me when I started making up my mind about this (and I'm not finished obviously because most of the important data just isn't there). So let's skip the "don't believe this or that guy" spiel about opinions, I try to form my own anyway. The facts and statements in that piece were interesting, that's why I linked it, not because of the author.
| First of all, you'd need access to every LAN and WAN through which data would travel. This is no mean feat in and of itself. The company I work for does real-time monitoring of network security in large organisations for counter-espionage purposes. We do that by fitting network taps at critical points inside infrastructure, and then drawing down data through our hardware where it is stored for analysis. This is complex and expensive- quite feasibly for a medium-sized company to spend millions of pounds on the hardware alone if they are operating across two or three sites. Therefore you are talking, say, £1.5m in hardware costs alone for ~8-10,000 machines monitored, with a data flow of about 100 million sessions per 24-hours across numerous protocols. Expand this across the millions of small-scale networks that exist in the US and your talking a huge financial cost. |
Well but you're now comparing networks or large organizations (corporations?) to the entire internet, or the "entire US internet". (Which is a problem like all "local" internets [=ones that can basically be switched off or disconnected from the oustide "internets" remotely; UPDATE:just like Syria showed once again today] btw, not to mention root DNS servers being pretty centralized blah... a little off topic at this moment)
I can't say much about the numbers you offered here, I don't know the capacities involved etc. so I can't say much else right here except remind of Schneier's post and the comments where they did the math as well, and linked a popular post that wants everyone to make sure to not refer to it as "news".
Generally you're right of course, it's not easy to tap into everything; but if you have the cooperation of the Telcos, like with Room 641A
as mentioned later, it's very very possible and has actually been done before. And by whom? AT&T & the NSA of course, the usual suspects. Those guys that suddenly had ex-employees going whistleblower on this sh*t after they had realized it was against the best interest of the American people and againts the constitution, the same people I mentioned in my earlier posts multiple times.
| Lets take the standard, Secure Socket Layer/TLS 1.2 protocol as an example... |
I've addressed this a little already; people like to act like SSL was secure but Türktrust, the case of the forged MS certificates, Abobe Certificates, Realtek certificates, etc. etc. etc. have again and again shown one thing: the CA system is fundamentally broken and is open to any adversary who truly wants "in". Oh another funny thing in that case: HTP took over the control over the root DNS for .edu domains
, I can only recommend reading their zine on their exploits (the latter pretty much literally; sh*t site is down atm, try later)
. They released the logins for most. If you get that level of control there isn't much you can't do, especially if you have some sort of country specific DNS like many of the arab countries did that were later critizied for shutting them off, blocking certain sites or redirecting them; oh and I remember one country actually even blocked all SSL traffic at one point.
The HTP case is also a fine example against something else you said: in some cases it's just beautiful to have tons of info to look through
In the context of our topic however it's true, you don't want all of the garbage but I'm sure the NSA has pretty good filtering techniques; actually another much older Schneier post on that topic- from 1999 (!), take note of what he mentions as possible for such agencies almost one and a half decades ago - gets into that more deeply; talking about the "ECHELON Technology" part: https://www.schneier...-gram-9912.html
A little quote:
| One of the major technological barriers to implementing ECHELON is automatic searching tools for voice communications. Computers need to "think" like humans when analyzing the often imperfect computer transcriptions of voice conversations.|
The patent claims that the NSA has solved this problem. First, a computer automatically assigns a label, or topic description, to raw data. This system is far more sophisticated than previous systems because it labels data based on meaning not on keywords.
Second, the patent includes an optional pre-processing step which cleans up text, much of which the agency appears to expect will come from human conversations. This pre-processing will remove what the patent calls "stutter phrases." These phrases "frequently occurs [sic] in text based on speech." The pre-processing step will also remove "obvious stop words" such as the article "the."
The invention is designed to sift through foreign language documents, either in text, or "where the text may be derived from speech and where the text may be in any language," in the words of the patent.
| SSL decryptors inside the network, like the boxes Sourcefire provide. |
Cool, I didn't know of these before.
| once you start doing pee-to-peer non-external-facing SSL connections, you are no longer communicating through the defined, clear area of a wide area network and therefore they cryptographic keys can't be acquired. |
Obviously, but you can't read people's thoughts either (yet, at least not officially - well, actually I remember reading they were officially working on it a few months ago but you know what I mean).
But if they go out you can listen to them. Same here. No problemo for usual intelligence gathering imho, it's always been like that.
| That's why secure peer-to-peer protocols and onion routing are so effective at encrypting data- because you can apply an additional layer of encryption to each transitional stage and therefore you need all the available private keys in order to decrypt the traffic. |
Hm, my impression was more that such networks relied on the idea that you couldn't tell which peer really requested what data but that may well be assured by such encryption.
I know what the (as usual) attention grabbing Gawker headline is screaming but there's nothing even remotely like that in there, lol. Read it yourself. They just say "check it out, Apple uses AES! So buy APPLE NAO BECAUSE APPPPPPPPLEEEEEEEEEEEEE!!!!!111111 Oh and guess who says it's totally secure? APPPPPLE!!!!! f*ck YEAH!!!" and they throw in a mention of the NSA approving AES for "top secret" info. First off after Wikileaks I think it became clear that "top secret" is by far not the most secretive designation there is, second again from Wikileaks and the things Manning has said that the "top secret" networks he had access to were wide open to hundreds of thousands of soldiers and people all over the world. On machines where they played Angry Birds, watched porn and heard Lady Gaga on. Security only helps until layer 8 comes into play.
| The second is economies of scale. We're at a point of seeing approximately 200 petabytes of conventional internet traffic every day. |
According to the Wikipedia numbers for the US internet backbone(s?!) you are probably correct
| A large percentage of this goes through US servers and service providers. Now, ISPs distribute their data handling capability- tens of thousands of terrabit-per-second junctions scattered across the country, but in order to hoover up all that data and to draw it off into a single repository you'd need an absurd collection infrastructure. We're talking about potentially hundreds of thousands, or even millions of individual taps. The US digital infrastructure just doesn't have the capability to effectively double already stretched capacity into transmitting captured data of this nature to a central repository. |
All this requires is the cooperation of ISPs (which have no choice in the USA) and money. IIRC the NSA is not exactly an agency with very limited funding...
| Another 20% is machine-based protocols that don't carry any human information, and can effectively be discarded. Why do you want to listen in on RPC and DHCP activity inside a network; it doesn't tell you much other than about the structure of the network. |
20% of all (American) internet traffic? Hard to believe for me but can't claim/prove it was impossible, do you have a link by chance?
| Why, also, would you want to listen in on the huge quantities of business-related data traffic which would have little to no intelligence purpose? But the question then becomes how do you determine what is what? ISPs provide for both businesses and individuals; MAC and IP addresses are just numerical representations of devices which don't actually possess any real identifying characteristics; the most you can determine from them in isolation would be the device manufacturer. |
Well, if you truly have capacity (of the lines, sniffing equipment or storage) issues you could just eliminate that from the get go by using the ISP's billing database to separate between business and private. Then if you have no businesses you want to keep an eye on just don't watch; but we've seen massive surveillance systems are often (ab-)used to spy on companies as well.
| Most externally facing IP addresses are dynamic, so the geography-specific data for them is only applicable for the time at which activity occurs from a single machine with a single dynamic address. |
Usually not even that; when I connect from my PC through my router to my ISP my outside IP resolves to the location of my ISP who owns the entire IP range along with several others, so there's no way of accurately resolving my location from my IP alone. Unless you're my ISP or have their DBs of course, then you can find out which customer logged in from which line under what IP (which addresses your next point after the above quote, it's very possible and all ISPs do it all the time, unless you mean something different).
And that's the point here; there's, as usual, one point where all the data is just readily available: the ISPs. And with POTS going the way of the dodo and almost all communication even today being done via TCP/IP that just means all the more data to collect, easily available for anyone who asks (and can produce a badge or letter to show).
| And that's quite aside from the fact that deep packet inspection capability- which the NSA already possesses- renders the requirement to store vast quantities of completely worthless data entirely null and void. |
DPI costs a lot of resources. Storing sh*t costs nothing once you have the space (yes, electricity, cooling etc, yeah; but nothing impossible). Besides you can still do DPI later on, filter stuff like I said. But first you just save it away, ask warez leechers, they usually follow the same principle; it's much much harder to check everything 100% reliably before it comes in than just letting everything come in and deleting what you definitely don't need later, all you need is the capability to capture and attribute (=cooperation of telcos/ISPs + specialized hardware), store (data centers like the one in Utah, cloud etc.) and analyze (DPI, speech recognition etc.). They have all they need so it seems naive to assume they won't do it, or aren't doing it.
Yet (like someone on the Schneier blog and I think here as well) one could make the argument that it's in the NSA's (and other similar Agencies') best interest for us to overestimate them but I strongly disagree, I very much doubt such people, who are no doubt intelligent if reckless and with no regard for the law or "what is right", would be so short sighted.
An adversary overestimating you is only a good thing as long as it doesn't come to a fight; then, you're f*cked, and the other guy will laugh his ass off. The other guy being the "bad guys" here of course, terrorists, pedophiles, Nazis, muslims, whatever the scapegoat du jour is that they're using to justify what they're doing against their own people.
| Cast your eyes back to your comments on the Utah Data Centre, and then consider two issues. One, that the data centre's primary purpose isn't to be a repository for all electronic communications in the US, but merely all electronic communications with an intelligence value. |
And who gets to define what has an "intelligence value"? Besides, how do you know that? And is "intelligence value" something static or can it change when the next nut kills people with a pressure cooker?
Seriously man, this is all very very unstable... let's not act like we know all the details here, neither of us do (no offense to you or your job but I assume the innards or the NSA are foreign to you as well).
| The NSA does much of their SIGINT work abroad, and the biggest target for data collection in the current economic and strategic climate is China. Also, whilst the trillion-terrabyte-storage capability sounds terrifying, it handily ignores the primary purpose behind having such computing power and storage capability- the breaking of complex cryptographic algorithms and cyphers. |
Dude pick one
First you said "the data centre's primary purpose isn't to be a repository for all electronic communications in the US, but merely all electronic communications with an intelligence value", then suddenly its "primary purpose behind having such computing power and storage capability [is] the breaking of complex cryptographic algorithms and cyphers".
I'll pretend there was no contradiction here and just answer the 2nd "primary purpose" you mentioned: are you actually proposing that the Utah data center will be used to STORE RAINBOW TABLES?
² Sorry but that's classic.
I remember back when the NSA was looking for someone to program their quantum computers; it said that the people looking to apply should just assume that such a system existed. Why do I mention this? That was at the very least 5 years ago. And I'm still far from convinced that quantum computers, in the hands of some of the best code breakers in the world, can't really speed up the process of cracking stuff.
So I doubt that's Bluffdale's purpose. That seems to be storage, and perhaps analysis of the data stored there, whatever it might be. And like I said I doubt they're planning to store Yottabytes worth of rainbow tables.
| The Wired Threat Levels article discussing the data centre goes into the value of this for intelligence gathering to a great degree. |
Ah ok but I still had to make the joke before
And you make it seem like the wired piece said the Utah data center would not be used to store comms which is not the case at all:
| Under construction by contractors with top-secret clearances, the blandly named Utah Data Center is being built for the National Security Agency. A project of immense secrecy, it is the final piece in a complex puzzle assembled over the past decade. Its purpose: to intercept, decipher, analyze, and store vast swaths of the world’s communications as they zap down from satellites and zip through the underground and undersea cables of international, foreign, and domestic networks. The heavily fortified $2 billion center should be up and running in September 2013. Flowing through its servers and routers and stored in near-bottomless databases will be all forms of communication, including the complete contents of private emails, cell phone calls, and Google searches, as well as all sorts of personal data trails—parking receipts, travel itineraries, bookstore purchases, and other digital “pocket litter.” It is, in some measure, the realization of the “total information awareness” program created during the first term of the Bush administration—an effort that was killed by Congress in 2003 after it caused an outcry over its potential for invading Americans’ privacy. |
------ this part was written yesterday; not that it makes a difference, just fyi; had to stop here.
| The idea of "grab everything and then sort it" is a very Cold-war mentality which works pretty well with electronic intelligence gathered specifically from targeted individuals and nation states but is fundamentally flawed when it comes to accurate and timely assessment and dissemination of intelligence collected from the open-source macrocosm that is wider society. My issue with your comments is that you appear to have formulated an argument which doesn't correlate logically with taught or operational intelligence practice. I questioned your understanding of the intelligence cycle and modern intelligence analysis techniques because your argument doesn't appear to acknowledge how the intelligence cycle works, nor does it accurately portray the most effective ways of gathering targeted, specific intelligence on subnational violent actors- who are the primary security threat to the US mainland. One of the strongest arguments against your thesis is that it doesn't actually make sense unless you suspend the accepted knowledge of how intelligence analysis works- instead it required a Hollywood cultured understanding of intelligence which doesn't actually reflect reality. |
Heh I read Schneier's movie plot threat contest too so don't lecture me
Seriously though, I understand where you're coming from but I think your scope is too narrow.
You're only applying current standards, completely dismissing any possibility of things being done that are beyond the standards. And that I simply find naive, no offense, in the context of agencies such as the NSA. Yes as I believe I said before we might well think too much of them sometimes but tbh I'd rather be "safe" than right.
| You have an ex-federal agent who says that the US has a great deal of capability in determining content and data from phone communications- big surprise. |
Well, I addressed this earlier. Other than that, saying "big surprise" doesn't make it any better imho.
| An AT&T operator saying that intelligence agencies are drawing down huge quantities of data to support investigative capabilities- hardly earth-shattering news. |
Not to you and me maybe, but I doubt it hurts to get the "general public" informed as well.
| And a noted constitutional lawyer with a history of speaking out against all kinds of perceived violation of civil liberties speculating about the capabilities of the US intelligence apparatus- how astonishing. None of these people have actually produced the same argument you have with the possible exception of the latter- but he talks more in terms of capability than actual active operational usage. You've made the fundamental mistake of extrapolating comments and extending them to what you see as a logical conclusion. |
Again I didn't mean to imply anything was a surprise; it's just nice to have evidence to back up things others might like to refer to as "conspiracy theories".
| I provided you with a list of other ways in which it would be perfectly possible to obtain the same data content without recording the entire conversation. How can they access this information? Well, the first port of call would probably be interrogating Tsarnaev's wife. |
Oh come on. They (and Clemente) were talking about finding out precisely what was said; that's something entirely
different than questioning his wife. If they meant that they could have said it, instead they said they'd try to find out exactly what was said - which is not
open to interpretation.
The only way to find out exactly what was said is to have it on record in some way, whether it's just snippets (as you seem to imply by specifically saying "the entire conversation"), a text log or the conversation saved directly.
| There are clearly numerous other ways in which this information could be obtained- external third parties, for instance. The actual phrasing of the comment itself seems to indicate investigators calling on members of the public and existing individuals with known relationships to Tsarnaev in order to obtain this information. |
Can you point me to anything at all that makes it sound like that?
| If they were interrogating a huge federal database of every conversation ever, why would they "want to find out" about it? Surely they'd already know? |
I assume "keeping" or "maintaining" instead of interrogating? Because that's my point. You always assume that they absolutely have to (for whatever reason...?) check everything right away. I say they don't - they save indiscriminately. And analyze later. That way they know they have the records, but they simply haven't looked into them yet. And suddenly that "finding out" thing makes sense again.
| I never said it was- that was your inference from my comments. I said it was the first time that it became part of a wide-ranging conspiracy theory that entailed the drawing down of all communications data. |
Hold it right there, Echelon is no conspiracy theory, it's well proven. And it did entail going through a whole lot of data.
| Are they? I'm pretty sure it hasn't been proven that the government are slurping up all communications data inside the US borders. |
Not conclusively yet, no; many things seem to point to it though. However you know as well as I do that many so called "conspiracy theories" are proven true sooner or later, I think I don't need to mention the likes of Wikileaks etc. here.
| Keyword analysis theory is pretty well known. Advertising is based almost solely on it. And yes, it is used in intelligence analysis. |
Well like I said "keyword theory" as you called it before only led to marketing related sites so this makes a lot of sense
f*ck, for a second I thought they had masturbation intelligence now.
| The people for whom interception may be worthwhile tend to use code and go to great lengths to obfuscate their activities, because they tend to have a relatively strong awareness of intelligence gathering capability and activity combined with the desire to not get caught. |
True. That doesn't stop crazy "security" fanatics from wanting to know everything about everyone. Facebook does it, why should governments be different?
| Keyword analysis has been utilised when codes used by organisations are known- one of the most famous ones in relation to violent Islamism is the discussion of weddings as a coded precursor to planning and perpetrating suicide bombings in particular- but this requires a pre-existing knowledge of the organisation which you don't possess by vacuuming up all the data available and running it through a slightly more intelligent version of Wolfram Alpha. |
<< implying by "sucking up everything" suddenly all other kinds of intelligence gathering had to be stopped because... well, makes for a nice point.
I never claimed anything like that. And if you have prior knowledge of who to look for an increased amount of data stored makes it no harder to look for; don't act like they don't keep databases, querying which easily leads to the precise HDD the selected conversation is stored on.
Plus you got lots of (temporarily at least) useless data that you can look through later if you have to, as seems to be the case with the Tsarnaevs.
| Intelligence analysis is an art. |
Nope, I'm not gonna say it
| It is scientific, but is not purely empirical. It is as much about cultural and human terrain analysis as it is anything else. Analysts work with an operational and strategic goal in mind- the strategic goal usually relating to the target on which intelligence is gathered, and the operational goal being the specifics of what ever macro-level analysis they are engaged in. |
No offense but I don't think I've ever heard more marketing bullsh*t speak in so few words. Are you a politician on the side?
I get your point but seriously.
| You cannot just simply give a team of analysts a huge quantity of data and say "find bad stuff". That's not how it works. |
Good thing I never claimed anything even close to that.
| It is fundamentally flawed to think that intelligence analysis is the art of finding a needle in a haystack- which is what you seem to be implying it consist of. |
Clear misunderstanding here; I never intended to claim that suddenly all US intelligence was only focused on getting everything. Nope, of course not. But I think it's becoming a part of the game, if it hasn't been for a while.
| As I've said numerous times, large quantities of data actually inhibit the ability to analyse them. |
Of course they do (in most cases, I have contrary examples, especially in console hacking, but that's a different topic so yeah). But I never claimed that they analyzed everything, only that they stored it; and for how long we can't even really guess of course.
| You might have to hunt around through numerous journals in order to find valid experiments and you might have to analyse and assess their worth in the context of your strategic goal, but you have a point of focus. This seems totally logical, does it not, and that's largely how intelligence analysis works. Now, try doing the same if your data set and related experiments are still present, but are stored on something like JSTOR for which you have full access but no search function. You've got to manually sift through all the articles on anthropology, Greek history and media management theory in order to find your data set and sources. Seems a bit silly, doesn't it? That's how you seem to imagine intelligence analysis works. |
As addressed right above, as well as before.
| Simple- by already possessing a decent understanding of the organisation which you are targeting, and knowing what keywords they are most likely to use. |
But my question was how one can gather
intelligence that way, not judge/analyze it. And that you still can't.
What you're describing is some sort of blind spear fishing; you know who to target and the phrases they use but you can't see them, so you just hope you pick up something with the keywords you're looking for. Is that really how intelligence operates? I don't mean to doubt it but then it sounds less interesting than tactics employed by chinese corp hackers tbh.
| Schneier, as in the cryptographer and security researcher Bruce Schneier? He's been pretty vocal in his statements that human-based macro-sphere applied intelligence analysis is vastly superior in all ways to automated intelligence trawling. |
Indeed but you seem to have gotten the impression I claimed the US suddenly switched tactics; but I don't think that, I just think they keep adding to their repertoire.