Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et
dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex
ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
mollit anim id est laborum.
The following grid shows the multiple editing capabilities built into dojox/grid/DataGrid.
Results2
Results3
Stack Group 2 Content in Editor
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et
dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex
ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
mollit anim id est laborum.
Anil Agarval Venture Finance ->
"Smithers, the monkeys are messing with the compiler again"
"I'll take care of it, Sir."
Photochrome Prints from the Lib. of Cong. [ca. 1890-1910] Bombay. Pydownee Street
Bombay. Girgaum Road
(Rights Advisory: No known restrictions on publication.)
This is a test
Tag cloud:
Toggle
Output:
http://www.imdb.com/chart/top (checked upto #70)
http://www.imdb.com/title/tt0301357/?ref_=nv_sr_1
https://images-na.ssl-images-amazon.com/images/M/MV5BMTI0MTg4NzI3M15BMl5BanBnXkFtZTcwOTE0MTUyMQ@@._V1_UY268_CR1,0,182,268_AL_.jpg
Good Bye Lenin! (2003)
(https://en.wikipedia.org/wiki/Ostalgie) nostalgia for aspects of life in East Germany.
Woman goes into coma just before the wall fall; when she wakes up her kids keep up a facade that the Soviet era still reigns.
[[NEED TO READ REVIEWS OF THESE BEFORE INCL. THEM]]
Crime, Mystery
http://www.imdb.com/title/tt0051201/
https://images-na.ssl-images-amazon.com/images/M/MV5BMTc0MjgyNTUyNF5BMl5BanBnXkFtZTcwNDQzMDg0Nw@@._V1_UX182_CR0,0,182,268_AL_.jpg
(1957) | Agatha Christie | Marlene Dietrich | A veteran British barrister must defend his client in a murder trial that has surprise after surprise.
http://www.imdb.com/title/tt0364569/
https://images-na.ssl-images-amazon.com/images/M/MV5BMTI3NTQyMzU5M15BMl5BanBnXkFtZTcwMTM2MjgyMQ@@._V1_UX182_CR0,0,182,268_AL_.jpg
2003 s. Korean | After being kidnapped and imprisoned for fifteen years, Oh Dae-Su is released, only to find that he must find his captor in five days.
http://www.imdb.com/title/tt0405094/
https://images-na.ssl-images-amazon.com/images/M/MV5BNDUzNjYwNDYyNl5BMl5BanBnXkFtZTcwNjU3ODQ0MQ@@._V1_UX182_CR0,0,182,268_AL_.jpg
The Lives of Others (2006) In 1984 East Berlin, an agent of the secret police, conducting surveillance on a writer and his lover, finds himself becoming increasingly absorbed by their lives.
http://www.imdb.com/title/tt0317248/
https://images-na.ssl-images-amazon.com/images/M/MV5BMjA4ODQ3ODkzNV5BMl5BanBnXkFtZTYwOTc4NDI3._V1_UX182_CR0,0,182,268_AL_.jpg
(2002)Two boys growing up in a violent neighborhood of Rio de Janeiro take different paths: one becomes a photographer, the other a drug dealer.
http://www.imdb.com/title/tt0114369/
https://images-na.ssl-images-amazon.com/images/M/MV5BMTQwNTU3MTE4NF5BMl5BanBnXkFtZTcwOTgxNDM2Mg@@._V1_UX182_CR0,0,182,268_AL_.jpg
Se7en (1995) Two detectives, a rookie and a veteran, hunt a serial killer who uses the seven deadly sins as his modus operandi.
Crime, Mystery
http://www.imdb.com/title/tt0114814/
https://images-na.ssl-images-amazon.com/images/M/MV5BMzI1MjI5MDQyOV5BMl5BanBnXkFtZTcwNzE4Mjg3NA@@._V1_UX182_CR0,0,182,268_AL_.jpg
A sole survivor tells of the twisty events leading up to a horrific gun battle on a boat, which begin when five criminals meet at a seemingly random police lineup.
Sci-Fi
http://www.imdb.com/title/tt0816692/
https://images-na.ssl-images-amazon.com/images/M/MV5BMjIxNTU4MzY4MF5BMl5BanBnXkFtZTgwMzM4ODI3MjE@._V1_UX182_CR0,0,182,268_AL_.jpg
Interstellar (2014) |Matthew McConaughey, Anne Hathaway, Jessica Chastain |A team of explorers travel through a wormhole in space in an attempt to ensure humanity's survival.
pOSS ALREADY IN DB.:
http://www.imdb.com/title/tt0407887/
https://images-na.ssl-images-amazon.com/images/M/MV5BMTI1MTY2OTIxNV5BMl5BanBnXkFtZTYwNjQ4NjY3._V1_UX182_CR0,0,182,268_AL_.jpg
An undercover cop and a mole in the police attempt to identify each other while infiltrating an Irish gang in South Boston.
Scorsese DiCaprio, Matt Damon, Jack Nicholson
http://www.imdb.com/title/tt2582802/
https://images-na.ssl-images-amazon.com/images/M/MV5BMTU4OTQ3MDUyMV5BMl5BanBnXkFtZTgwOTA2MjU0MjE@._V1_SX214_AL_.jpg
2014 (Philippines) A promising young drummer enrolls at a cut-throat music conservatory where his dreams of greatness are mentored by an instructor who will stop at nothing to realize a student's potential.
http://www.imdb.com/title/tt0043014/
https://images-na.ssl-images-amazon.com/images/M/MV5BMTc3NDYzODAwNV5BMl5BanBnXkFtZTgwODg1MTczMTE@._V1_UX182_CR0,0,182,268_AL_.jpg
(1950) A hack screenwriter writes a screenplay for a former silent-film star who has faded into Hollywood obscurity. dIR: Billy Wilder
[Please select tags:]
Notes:(last updated 2016.12.07)
NoteBook
Add Org module based off treeGrid
Add AdminMode on/off w/pwd triggered by acc expand/coll <
> hide toolbar buttons(fwdStack) if off
Need 2nd dlgBox size (larger) for, eg, LinerNotes
IF mBox has text larger ~len, put "..." clickable into mAlert/mDlgBox
Add "reading" to rec accordian; move stuff from misc/Other
Film
MOVIE UPDATES CONTINUED @ NatlBoardAwards http://www.notcoming.com/ Midnight Eye - Visions of Japanese cinema Bottom of this page has links to desi Film Awd winners : https://en.wikipedia.org/wiki/Shyamchi_Aai_(film)
https://www.criterion.com/explore/149-yakuza
https://www.criterion.com/explore/19-samurai-cinema
20160930mpt yakuza.txt Liner Notes:
Film sections to add:
Historical (Bios of Napoleon/Monet/etc.) Music - Live shows/concerts Theater
-Categ & good liner notes (thoughtful/th~provoking articles) for Maigret Movies - Woody Allen - Spike Lee -
- Wimsey, Astaire, Hitchcock, Tarantino, Mel Brooks Spl Interest Categories: Literary / Art etc. (about, incl. biopix) New root categ for Spotlight? If using dojo:query; do so with a sep section for LinerNotes as well (They can have keywords too)
Notes on SELECTION: There has to be a system (V imp.) otherwise results in inequity
20161206mpt test tree Shell 8pm (Using dojo.tree to represent store data; with put/etc.,
dynamically updating dojo.tree widget using dojo.Observable (ObjectStoreModel)
Not as Spaniards, but as traitors, thieves, and murderers...a NewYorker visits St.Augustine, FL Travels through Switz... pursuing, like the swallow, the inconstant summer of delight and beauty which invests this visible world (the Shelleys)
Last stable bootcrap Launcher (8/15/16 or thereabouts)
(for later?)
(srch results for idler) http://tinyurl.com/jy5vb79
* insert col-sm & col-md tags
* preview/test in handheld brosers online
lookup public todo app w/export data fmt outlk
Left to add:
Blogs: Man~ AND -anglophile =epicurean -single malt avail =alfresco sm frndly opts
TV Mini-Series (72-75) | ~10 episodes | Ian Carmichael
TV Mini-Series (72-75) | ~10 episodes | Edward Petherbridge...more calm, solemn and had a stiff upper lip, subtly downplaying many of the character's eccentricities.
(1977) John Cleese | The grandson of the world's first and foremost consulting detective and his bumbling, bionic sidekick attempt to catch the only living descendant of Professor Moriarty.
(2001-2) 27 episodes | Highly rated A&E Series
Ed McBain's 87th Precinct: 3 movies | Lightning, Ice, Heatwave (1995-7)
(1962) Matthau is a syndicate mobster angry @ streak winner Dean Martin who's being set up.
(1963) Gangster Matthau is mad when movie w/his stripper girlfriend bombs; producer/writer run away to "island of love".
(1963) Paris | Hepburn, Cary Grant | Mathhau is a cop | film is in pub. dom; can watch it @ wikipedia
(1965) A biochemist suffers memory lapses, finds bad guys after him, and has no idea why. Gregory Peck, Matthau as investigator.
(1966) Matthau plays shyster lawyer | Dir. Bill Wilder (Irma la D)
(1968) farce abt the adventures of its naive heroine. Brando, Richard Burton, Coburn, Matthau, Ringo Starr Many cameos incl. Sugar Ray Robinson.
(1969) a strong-willed matchmaker(Streisand) travels to Yonkers, to find a match for "well-known unmarried half-a-millionaire".
(1969) Matthau breaks relationship lying that he is married. Also has Ingrid Bergman, Goldie Hawn
unsuccessful, now considered cult classic. (1971) playboy Matthau has 6 wks to find a rich bride & repay loan, or forfeit all property to uncle
(1972) Matthau plays a confirmed bachelor w/ eccentric habits. *s Carol Burnett,
(1973) In San Fran, Matthau investigates when a busload of passengers are gunned down & killed.
(1973) Matthau, a former stunt pilot, holds up bank in sleepy NM town.
(1974) group of criminals takes passengers hostage inside a NYC Subway car for ransom.
(1974) Lemmon is reporter @ Chicago paper & Matthau is ruthless editor | dir by Billy Wilder (Irm la d)
(1978) Similar to 'Plaza Suite', based on play by Neil Simon. Visitors from London/Philly/Chicago.
(1980) Matthau is a renegade CIA agent abt to expose the cia/kgb
(1981) Matthau is hitman stuck in hotel w/Lemmon who wants to kill himself.| dir by Billy Wilder (Irm la d)
(1983) in NY, Matthau's gas station is accidentally blown up by Robin WIlliams "unconventional humour" people thot the two didn't gel tog.
(1996) 2 old fogies meet in NYC Central Park. Film adaptation based on tony-awd winning play
(1997) the pair are dance hosts on a cruise ship trying to seduce heiresses.
(1998) a sequel, d'you believe it?
(1959) 2 musicians who dress in drag in order to escape from mafia gangsters | dir. Billy Wilder (Irma la D)
(1960) Lemmon's Upper WestSide apt is used by 4 mngrs for trysts | dir. Billy Wilder (Irma la D)
(1972) Lemmon's dad dies in Naples but not alone (w/mistress) bodies disappear, etc. | dir. Billy Wilder (Irma la D)
(1965) Has to be a FREAKING good movie (plus, look at the date) | Basil Fawlty claims to have seen it six times,
(1960) Ocean's 11 is a heist film centered on a series of Vegas casino robberies
(1962) remake of Gunga Din set in the American West. Last film to feature all 5 due to Sinatra's falling out with Lawford.
(1964 ) the film transplants the Robin Hood legend to a 1930s Chicago gangster setting
(1947) (Frank Sinatra and Peter Lawford)
(1956) a gambling rancher (Dailey) discovers that all he has to do to win at roulette is take hold of ballerina Charisse's hand
(1958) a troubled Army veteran and author who returns to his Ill. town after 16 yr, to the chagrin of his wealthy, social-climbing brother.
(1959) Charles Bronson, Steve McQueen, Sinatra/Lawford fighting the Japanese in Burma in World War II despite a lack of support from their commanders
(1963) based on the play of the same name by Neil Simon.
(1963) exiled American in Rome, rescues a young Sicilian outlaw, from the police & turns him into Johnny Cool.| Telly Savalas, Sammy Davis, Jr.
(1963) thug Charles Bronson & a cameo by the Three Stooges
(1965) a businessman's wife who ends up divorced by mistake and then married to his best friend by an even bigger mistake
(1966) story of a self-destructive jazz musician (Sammy Davis) Also features Louis Armstrong, Mel Torme & Frank Sinatra
(1966) comedy/satire abt a love-triangle made more complex by an Indian maiden & a Comanche raiding party
(1968) At a nightclub in Swinging London, a dying Asian girl sets off plot to overthrow the British govt
(1970) Salt & Peppar redux
(2012) Western dir Quentin Tarantino,
(2012) criminal syndicates use contracted killers called "loopers" to kill victims sent through time travel.
(2012) based on 96 murder of 80yr-old millionaire in TX by her 39yr-old companion.
(2012) cops & bloods in LA police procedural
(2012) skateboarding subculture movie in LA
(2013) Sandra Bullock % George Clooney are stranded in space
(2013) American war film (Afghanistan) based on true-life act.
(2013) crime drama Hugh Jackman
(2013) Emma Thompson as author (Mary Poppins) and Tom Hanks as Walt Disney
(2013) Ben Stiller / life at Life magazine
(2013) biographical black comedy crime film directed by Martin Scorsese
(2013) based on the life story of the Wing Chun grandmaster Ip Man
(2013) crime prison drama
(2013) crime drama NY motorcycle stuntman working in a traveling act for state fairs.
(2014) mob movie based in 1981 New York
(2014) satirical black comedy-drama film dir by Alejandro G. Innarritu.
(2014) tank crews in Nazi Germany
(2014) psychological thriller Ben Afflek's wife is missing
(2014) Alan Turing, who decrypted German intelligence codes during World War II. Cumberbatch
(2014) thief shoots live footage of accidents/crimes in LA, selling content to news channel
(2014) husband is believed to have prioritized his own escape frm avalanche over safety of family.
(2014) current life in small-town N. Russia nr Finland
(2013) British prison with some of the country's most violent criminals
2013 thriller film funded via crowdfunding
2014 spy thriller - chechnya, germany, le carre
(2014) BioPic won a few awards
(2013) S Korean-Czech science fiction thriller film based on French graphic novel.
Cafe Society (2016)Bronx native moves to Hollywood, where he falls in love with the secretary of his powerful uncle, an agent to the stars. After returning to New York, he is swept up in the vibrant world of high society nightclub life.
Irrational Man (2015) tormented philosophy professor finds a will to live when he commits an existential act.
Magic in the Moonlight (2014)A romantic comedy about an Englishman brought in to help unmask a possible swindle. Personal and professional complications ensue.
Midnight in Paris (2011)While on a trip to Paris with his fiance's family, a nostalgic screenwriter finds himself mysteriously going back to the 1920s everyday at midnight.
The Curse of the Jade Scorpion (2001)An insurance investigator and an efficency expert who hate each other are both hypnotized by a crooked hypnotist with a jade scorpion into stealing jewels.
Small Time Crooks (2000)A loser of a crook and his wife strike it rich when a botched bank job's cover business becomes a spectacular success.
Sweet and Lowdown (1999) 1930s, jazz guitarist idolizes Django Reinhardt, faces gangsters, falls in love w/a mute woman.
Mighty Aphrodite (1995) When he discovers his adopted son is a genius, a New York sportswriter seeks out the boy's birth mother: a ditzy porn star and prostitute.
Bullets Over Broadway (1994) In New York in 1928, a struggling playwright is forced to cast a mobster's talentless girlfriend in his latest drama in order to get it produced.
Shadows and Fog (1991) Woody Allen, Mia Farrow With a serial strangler on the loose, a bookkeeper wanders around town searching for the vigilante group intent on catching the killer.
Crimes and Misdemeanors (1989) An ophthalmologist's mistress threatens to reveal their affair to his wife, while a married documentary filmmaker is infatuated by another woman.
The Purple Rose of Cairo (1985) In New Jersey in 1935, a movie character walks off the screen and into the real world.
Broadway Danny Rose (1984) In his attempts to reconcile a lounge singer with his mistress, a hapless talent agent is mistaken as her lover by a jealous gangster.
Manhattan (1979) life of a divorced television writer dating a teenage girl is further complicated when he falls in love with his best friend's mistress.
Love and Death (1975)In czarist Russia, a neurotic soldier and his distant cousin formulate a plot to assassinate Napoleon.
Bananas (1971) When a bumbling New Yorker is dumped by his activist girlfriend, he travels to a tiny Latin American nation and becomes involved in its latest rebellion.
Pussycat, Pussycat, I Love You (1970)An American playwright living in Rome consults a quack psychiatrist to combat his fears of balding and save his failing marriage.
Take the Money and Run (1969)The life and times of Virgil Starkwell, inept bank robber.
What's New Pussycat (1965) Peter Sellers, Peter O'TooleA playboy who refuses to give up his hedonistic lifestyle to settle down and marry his true love seeks help from a demented psychoanalyst who is having romantic problems of his own.
Jaanisaar (2015)(from Muzaffar Ali, dir of Umrao Jaan) courtesan from Avadh, and a prince brought up in England, set in 1877...introduces Janaab Nazeer Akbarabadi, known Urdu poems for both Ali/Krishna
Lakhon Ki Baat (1984) Sanjeev Kumar, Farooq Shaikh. Photographer & pals decide to sue.
Ab Ayega Mazaa (1984) Farooq Shaikh.
Uddhar (1986) Farooq Shaikh, Supriya Pathak
Anjuman (1986) music by Khayyam, dir Muzaffar Ali |Shabana Azmi, Farooq Shaikh, R Hattangadi Set in Lucknow, it deals with exploitation of women and problems of local "chikan" embroidery workers. three songs by Shabana Azmi.
Ek Pal (1986) Farooq Shaikh, Shabana Azmi, Naseeruddin Shah,
Dir Muzaffar Ali. Farooq Shaikh, Smita Patil. Lucknowite re-locates to Bombay. Has "Seene Mein Jalan...Toofaan"
(2010) Farooq Shaikh. thriller based on a real-life incident of a woman who hit a man, drove home & parked the car w/the man wedged halfway thro' the windshield.
Detective Byomkesh Bakshy! (2015)Dir Dibakar Banerjee The origin story of famous Bengali sleuth created by Saradindu Bandopadhyay.
Khosla Ka Ghosla! (2006)Dir Dibakar Banerjee Delhi based retired man tries to get his land back from a swindler.
Titli (2014) a violent car-jacking brotherhood in Delhi
Talvar (2015) An experienced investigator confronts several conflicting theories about the perpetrators of a violent double homicide.
Nikaah (1982) Salma Agha
(2014)Helen Mirren, Om Puri. The Kadam family leaves India for France where they open a restaurant across from a Michelin-starred eatery.
36 Ghante (1974) Raaj Kumar, Sunil Dutt Three jailed convicts break out of prison and take over the household of Editor Ashok Rai.
Baaz (1953) Guru Dutt, Johnny Walker2 desi girls fight back in Portuguese-ruled Malabar region in India
Mr. & Mrs. '55 (1955) Guru Dutt, Madhubala naive heiress forced into 'marriage of convenience' w/unemployed cartoonist
12 O'Clock (1958) Guru Dutt, Waheeda Rehman (!) Maya is found dead in the first class compartment at a railway terminus in Bombay
Chaudvin Ka chand (1961) Guru Dutt, Waheeda Rehman (!) love triangle in Lucknow
Sahib Bibi Aur Ghulam (1962) Guru Dutt, Waheeda Rehman (!)servant Bhootnath gets close to the wife of his employer, and narrates her story.
Has "Bhanwara Bada Naadaan".
Bharosa (1963) Guru Dutt, Mehmood
Suhagan (1964) Guru Dutt, affair between a professor and a student
Sanjh Aur Savera (1964) Guru Dutt, Dir. Hr Mukherjee
Maigret tend un piage (1958) Gabin
Maigret et l'affaire Saint-Fiacre (1959) Gabin
Maigret voit rouge (1963) Gabin
Maigret(TV series|54 episodes|91-05)...Certes, le grain de l'image a vieilli,
les lumieres ou la musique ne sont pas toujours extraordinaires,
mais cette ambiance est unique.
Maigret Sets a Trap (2016) TV Movie Rowan Atkinson (reviewer) ...The scenery, photography, costumes, props and direction were beautiful and the movie was 100% traditional 'Noir' in every popular sense of the film world. The shadows and lighting, cars, streets, the brown... (lots of brown!), even the smoking and the hats... all Film Noir!
(another rev) ...Atkinson's calm and soft-speaking portrayal is the first proper portrayal of Maigret I've ever seen (all the others were a bit too flashy for my taste).
Maigret's Dead Man (2016) TV Movie Rowan Atkinson
Maigret(TV series|53 episodes|59-63) Rupert Davies
(review)...captured mood and style of Simenon's books
...captured the atmosphere of Simenon's Paris perfectly with its Citroen Tractions, cobbled streets and bistros.
...Half the UK population flocked home to see it each week.
Les enquites du commissaire Maigret (TV Series|88 episodes|67-90) ...has to be good; prod. while Simenon was alive.
Maigret (1988) Richard Harris's version; if likeable they made a TV series w/him too
S Korean, (1989)The title itself is a koan. "... better than any previous film of this type."
2009 (Japan) Dogen Zenji
S Korean, Spring, Summer, Fall, Winter... and Spring (2003)
set in Ladakh, Himalayas
Milarepa (2006) Photographed in the stunning Lahaul-Spiti region of Northern India
(2005) Document. A journey exploring the practices of Chinese hermits living in the Zhongnan Mountains.
"...a clever little Indie about the American zen experience" (Comedy)
The Sea That Thinks (2000) comme "The Mind's Eye"
Documentary: Rinpoche (2011) stars Ginsberg
Document.: 1998 (Switzerland) Steps of mindfulness
Cave in the Snow (2002) 52min | Documentary | Australia
Around the World in 80 Gardens: China and Japan | Documentary 2008
(1958) Released from prison, a gangster (Michitaro Mizushima) retrieves diamonds sought by fellow yakuza.
(1960) A prison truck is assaulted and the two convicts inside are murdered.
(1960)Two reporters of divergent morals investigate a drug ring, delving deeper into the underworld in the process.
(1960)young rebel Juro has to deal with an environment of crime and prostitution
(1961)A Hell of a Guy | Muteppo-daisho (original title)
(1961) The Man with the Hollow-Tip Bullets |
Sandanju no otoko (original title)
Detective Bureau 2-3: Go to Hell Bastards (1963)
Kutabare akuta-domo - Tantei jimusho 23 (original title)
Youth of the Beast (1963) Yaja no seishun (original title)
Akutaro (1963) The Bastard
Nikutai no mon (1964)Gate of Flesh
Irezumi ichidai (1965) Tattooed Life
Tokyo nagaremono (1966) Tokyo Drifter
J Shishido,(1967)
(2001) Pisutoru opera (original title)
(1985)... possibly the ne plus ultra of Seijun Suzuki provocations, and must be seen to be believed... Made entirely in Japan, and with dodgy English-language acting by both the Japanese and American performers, it's an explosion of color and anachronistic costumes and hairstyles, with Kimura's set design repurposing a Japanese amusement park to approximate the West Coast of the United States.
Kanta mushuku (1963)
Water (2005), India in 1938, movie follows Fire(1996) & Earth(1998). Deepa Mehta/Anurag Kashyap
(2006) based upon J. Lahiri's bk / Mira Nair
2008 British drama film
2009 science fiction thriller aliens in '82 Jo'burg
1996 Wes Anderson | Owen & Luke Wilson
1998 Wes Anderson | Bill Murray
2001 Wes Anderson | Gene Hackman Ben Stiller
2004 Wes Anderson | Owen Wilson hunt for shark
2007 Wes Anderson | Owen Wilson train voyage across India
(1962)
(1966) Believing himself to be a suspect for embezzlement in British India, a man must testify against non-violent freedom-fighters.
Biwi Aur Makan (1966) 4 guys come to Bombay & 2 of 'em dress in drag to get accomodation
(1971) Bombay, Hanging gardens etc.
(1972)family has a reputation of not having any cook last long
Sabse Bada Sukh (1972) Village born Lalloo re-locates to Bombay
(1973) Delhi, Calcutta
(1975)new husband plays a practical joke on wife's family
Alaap (1977)
Chaitali (1975) Dharmendra, based on novel of the same name by famed Bengali writer Ashapoorna Devi
1976 - Keshto
(1980)
Naukri (1978)A.K. Hangal Rajesh Khanna Raj Kapoor
(1979) Hangal, Keshto, bachchan
(1981) Amol Palekar, Swaroop Sampat, Utpal Dutt |
Achha Bura (1983) Amjad Khan, Ranjeet, Raj Babbar
(1983) Farooq Shaikh, Deepti Naval, Utpal Dutt |
(1998)
(2007) flooding the Reich w/fake notes
(2007) bio-Piaf
set in Communist Romania ~ final yrs of Ceau?escu era
Spy movie set in Hong Kong in 1938 and in Shanghai in 1942
2007 Turkish-German drama
2007 French film French slang for "trashy novel fr. train station"
Viennese ex-con & Ukrainian prostitute - bank robbery.
a 2008 Iranian movie man works @ ostrich farm; fired when an ostrich escapes
2008 Turkish film Istanbul
2009 black-and-white German-language drama the roots of evil
2009 French prison drama-crime film: the Corsicans vs the Muslims.
2009 German comedy film re: Greek tavern in Hamburg
cop-shooters in '88 Melbourne, Austr.
2009 British drama film / 15yr old in an East London council estate
2010 pol thriller film directed by Roman Polanski.
explores ennui among Hollywood stars / Chateau Marmont / S. coppola
3 generations of middle-class Taiwanese family in Taipei.
Hong Kong in 1962, v. highly rated movie
Hyeol-eui-noo (2005) Blood Rain | Dae-seung Kim
(1974) A Delhi-based woman in a steady relationship gets an old Mumbai flame back in her life.
(1975) "established Amol Palekar as having an uncommon comic talent for playing mousy characters..."
(1976) Bombay Zarina Wahab
(1977) Smita Patil, Dina Pathak, Naseeruddin Shah |Girish Karnad (screenplay)
Shootout at Wadala (2013) The first-ever registered encounter by the Mumbai Police, which took place on November 1, 1982. Based on a true story.
2013 Filmfare_Critics_Award_for_Best_Film - Gangs of Wasseypur - Anurag Kashyap
Once Upon a Time in Mumbai (2010) A smuggler rises to power in in 1970s Mumbai, a younger gangster seeks to overthrow him, and a police officer is caught in the middle.
Once Upon a Time in Mumbai Dobaara! (2013)
Dhoom 3- 2013
Chennai Express - 2013 about a man's journey from Mumbai to Rameswaram, and what happens along the way after he falls in love with the daughter of a local don.
Bajirao Mastani 2015 Indian epic historical romance film directed by Sanjay Leela Bhansali
Agantuk (1991) The Stranger(Saty. Ray, Utpal Dutt) Comedy
Aranyer Din Ratri - Days/Nights in the Forest (1970) S. Ray | Calcutta city slickers head out into wilderness
Xagoroloi Bohudoor (Sea v. far) 1995 award winning Assamese boatman loses biz when govt builds bridge
(1953) Marathi
Kathapurushan (1996) awd-winning film abt char initially a Gandhian eventually a Marxist
(The Red Door) 1997 Bengali Kolkotta dentist reflects on his life vs his drivers, lal door's from his childhood a ala Orwell.
1999 Malayalee award-winning film re: Kathak dancer plays Arjun upp-class audience girl falls for him & they have child but he finds she loved the stage Arjun not the real person.
Dhoom 3- 2013 Aamir Khan, Katrina Kaif, Abhishek Bachchan
Maqbool (2003) Irrfan Khan, Naseeruddin Shah(Dir Anurag Kashyap) 'Macbeth' in Mumbai underworld
Garam Masala (2006) Comedy
Jodhaa Akbar (2008) (Hritik Rosh/Aishw) great Mughal emperor, Akbar, and a Rajput princess, Jodha
Mumbai Meri Jaan (2008) follows several interconnected stories re:'06 train bombings
Oye Lucky! Lucky Oye! (2009) based on true story of Lucky Singh, charismatic conman frm Delhi
3 Idiots (2009) College days in Delhi's Eng. College
Firaaq (2010) Muslims in Gujarat after riots
Dabangg (2010) A corrupt police officer (later 2 sequels)
Zindagi Na Milegi Dobara (2011) Hrithik Roshan, Farhan Akhtar, Abhay Deol decide to turn their fantasy vacation into reality
Rowdy Rathore (2012) (Sanjay Leela Bhansali) A con man uncovers a deadly secret and must save ppl from the mob (Dir. Prabhu Deva; became one of the highest-grossing Indian films)
Ankhon Dekhi (2015) Comedy, drama "out of the box"
Gabbar is Back (2015) (Sanjay Leela Bhansali) Gabbar Singh Rajput (Akshay Kumar) creates his own vigilante military network
Visaaranai (2015) (Dir Anurag Kashyap) Labourers tortured by police to confess to a theft they have not committed.
Ae Dil Hai Mushkil (2017) Karan Johar, 'slick actors/locations'
Padmaavat (2018) (Sanjay Leela Bhansali) Set in medieval Rajasthan
Hindi Medium (2018) Irrfan Khan, couple from Chandni Chowk aspire to give their daughter the best education
Gully Boy (2019) lives of street rappers in Mumbai; Zoya Akhtar
Aparajito (1956) Satyajit Ray a boy leaves home to study in Calcutta
Garm Hava (1974) Balraj Sahni, A.K. Hangal partition: considered 'a v influential film'
Nishant (1975) Dir. Shyam Benegal Girish Karnad, Shabana Azmi, Naseeruddin Shah village Zamindars
Junoon (1979) Dir. Shyam Benegal circa 1857 Indian Nawab desires to wed a young Anglo-Indian woman Shashi Kapoor, Naseeruddin
Sadgati (1981) Dir. Satyajit Ray premchand's story re: Untouchable shoemender Om Puri
Bhavni Bhavai (1981) Dir. Ketan Mehta atrocities on lower caste ppl
Utsav (1984) Dir. Karnad; Rekha, Kharbanda, Amjad Khan 200 BC courtesan Vasantsena
Mohan Joshi Haazir Ho! (1984) Dir. Saeed Akhtar Mirza Dina Pathak, old couple sue landlord of collapsing apt
Holi (1985) Dir. Ketan Mehta Naseeruddin S, Om Puri, hostel life
Pestonjee (1988) Nas. Shah, Shab. Azmi, Mumbai Parsee community in the '50s and 60s
Hero Hiralal (1988) Dir. Ketan Mehta Naseeruddin S, rickshaw driver is a hero
Salim Langde Pe Mat Ro (1989) unemployed youth aspires to big crime but mends his ways
Kamla Ki Maut (1989) Dir.Basu Chatterjee Pankaj Kapur, Irrfan
Suraj Ka Satvan Ghoda (1992) '1 of Shyam Benegal's most abstract films ever'... stories of meeting three women in diff life stages. Also known for its subversive take on the "Devdas" syndrome
Patang (1993) Dir. Goutam Ghose S Azmi, Om Puri life of people in illegal slums in Bihar
Padma Nadir Majhi (Bengali, 1993) Dir. Goutam Ghose pre-partition man wants to establish utopia in Padma delta (Bangladesh)
Maya (1993) Dir. Ketan Mehta based on "Madame Bovary"
Droh Kaal (1994) Dir. Govind Nihalani Om Puri, Naseeruddin Shah Undercover cops infiltrate a terrorist organisation (tr. Times of Treason)
Yatra (2006) Dir. Goutam Ghose Rekha, N Patekar, D Naval Writer meets filmMaker & discuss prostitute
Dharm (2007) Pankaj Kapur, Supriya Pathak a priest's wife brings home an orphan boy. Shot in Varanasi.
Rang Rasiya (2008) Dir. Ketan Mehta biopic 19th century Indian painter Raja Ravi Varma
Kaalbela (2009) Dir. Goutam Ghose Naxalite movement in 70s Calcutta
Raahgir (2019) Dir. Goutam Ghose encounters between poverty-stricken travellers
The first American co-production for the cult Japanese auteur "Beat" Takeshi Kitano. Brother, shot mostly in the States and delivered mainly in English, could be the biggest launchpad yet for a director and actor who remains almost unknown in Britain, despite being a megastar at home. This is a gangster film like no other, in which the violence never seems gratuitous or glamorised because of the dreamlike surrealism of the camerawork, and the wit and creativity of writing and performance. As a performer, Kitano manages to convey total menace with an acting style so minimal that by the end you almost forget that he's barely finished a sentence. Includes a 48-minute documentary on the film-maker, and cast and crew interviews.
More wondrous sights are on display in Hana-Bi, the most thoughtful film yet from the Japanese cult director Takeshi Kitano. The title means Fireworks, though the hyphen pointedly splits up the title into the Japanese words for flower and fire. Flower and fire, beauty and violence: key ingredients in Kitano's world, where gangster bloodshed is regularly interrupted by lyrical interludes, often set by the sea (as in Sonatine). The current film refines his style to the nth degree; dialogue is so sparse that you might almost be watching a silent film, and the elliptical editing whisks us, at times disconcertingly, between different points in time.
Kitano's characters have never been chatterboxes, but detective Nisi (Kitano himself, under his acting name Beat Takeshi) scales a new peak of silence. His wife is diagnosed with a terminal illness. His colleague Horibe is shot during a stakeout, and becomes confined to a wheelchair. Feelings hidden under dark glasses and tight lips, Nisi leaves the force and robs a bank to finance, among other things, a last trip with his wife and the supply of art materials to Horibe, who has taken up painting. But a blunt rehearsal of the plot can give no indication of the film's visual power.
This is Kitano's most personal work. He himself escaped death four years ago in a bike accident, and he is the painter of Horibe's artwork, bizarre amalgams of flowers and animals. The director's oblique manner needs an audience on the ball; but Kitano can make the most taciturn gaze speak volumes, and as the jigsaw builds there is never a doubt about the emotional force of this meditative tale of flowers and fire, of life and beauty in the face of death.
To anyone unfamiliar with the Japanese director and media star Takeshi Kitano, his film Sonatine may be something of a mystery. One minute, bullets fly and bodies fall with sudden, icy brutality, as bands of gangsters fight for supremacy in Okinawa. The next, those persons still left alive lark around with guns, seaweed and a soft-drink can on a beautiful beach, under a piercing blue sky.
Whatever the mood, the leading character Murakawa, played by the director under his acting nickname ``Beat" Takeshi, adopts a frozen face that suggests a huge existential death wish. Is this a yakuza thriller, an art movie divertissement, or what?
Essentially, what Takeshi has done is to yoke together two separate styles pursued in the three previous films since this extraordinary character actor, stand-up comic and television personality took up directing in 1989. The idyll by the beach recalls the tender, poetic simplicity of the mesmerising A Scene at the Sea. The quirky brand of bloodshed harks back to Violent Cop and Boiling Point, films whose casual violence can churn even an experienced stomach.
The result is a bizarre and awkward hybrid; for Western tastes, sensitivity and violence make strange bedfellows. But Takeshi's strong visual eye and elliptical manner (he serves, importantly, as his own editor) always keep our eyes glued to the screen. Sonatine takes you by surprise sometimes pleasantly, sometimes not: a valuable asset in a world dominated by Identikit films.
(2005 London Film Festival)
BLOOD AND BONES Beat Takeshi gives one of his most powerful performances in this portrait of a brutal businessman and his family.
Meanwhile, non-animation Japanese directors are breathing new life into tired Hollywood genres. Violent Cop was shown in Britain on limited release last month to much excitement. Its handsome director and star, ``Beat" Takeshi, was seen by reviewers as ``the next international cult figure" and as ``Japan's answer to Clint Eastwood". The film is certainly as violent as its title suggests, but, unlike the recent Hollywood police movies, you are kept in suspense rather than in a state of deja vu.
From 2011 The Daily Beast:
In 2009, Tadamasa Goto, the former Yamaguchi-gumi (Japan's largest
crime group) gang boss, wrote extensively in his autobiography,
Habakarinagara, of his connections to Japan's entertainment
industry, and justifies the attack his members made on a
famous Japanese film director, Juzo Itami, who dared to depict
the yakuza unfavorably in his dark comedy
The Gentle Art of Japanese Extortion.
Kitano's last film, Outrage is the most accurate portrayal
of the modern Yakuza ever. The film is a diatribe against
them, showing how the traditional yakuza that upheld some
primitive code of honor end up being betrayed or killed.
The ones that survive are those who are ruthless,
backstabbing businessmen - Goldman Sachs with guns.
If you want to understand the modern yakuza, Outrage
is a fantastic introductory course...
Bomber Jacket Tyler Folio
Custom Index NoteCards
Lincoln Vintage Metalsmith Tornado Rollerball Pen
Aurora Gemstone Rollerball Pen
Waterford Pallas Rollerball Pen
Parker Sonnet Cisele Rollerball Pen
Faber-Castell e-Motion Pure Black Rollerball Pen
ACME Rhapsody Rollerball Pen
Single-Sheet Cutters
Article: Some words about Khayaal some Indian classical music on EP/LP
201608140319mpt Yahoo Search result for urdu+ghazal+ghulam+ali+ghalib
http://www.avsforum.com/
http://www.klipsch.com/
http://www.polkaudio.com/
http://definitivetechnology.com/
http://www.martinlogan.com/
https://www.cambridgeaudio.com/
http://www.bang- olufsen.com/da/collection/speakers/beolab-20
http://www.jbl.com/loudspeakers/STUDIO +L890.html?dwvar_STUDIO%20L890_color=Black#start=1
http://www.paradigm.com/products- current/type=tower/model=prestige-95f/page=overview
http://kef.com/html/us/index.html
http://www.bowers- wilkins.com/Speakers/Home_Audio/600_Series/683.html
http://www.bostonacoustics.com/
Sony Home Cinema Projector - VPL- HW40ES
http://www.amazon.com/Koss-PortaPro- Headphones-with-Case/dp/B00001P4ZH/ref=sr_1_1?s=electronics&ie=UTF8&qid=1464305449&sr=1-1&keywords=Koss+PortaPro+Headphones+with+Case Stereophile
http://www.theabsolutesound.com/
http://www.soundandvision.com/ Radeon Sennheiser over-the-ear wireless Headphones
http://www.tomshardware.com/s/pc+4k/
ASUS Sound Card Essence STX II
Srch for "Essence ASUS headphone movie 2016" tinyurl.com/zrtpnn2
http://www.dansdata.com/spkvshead.htm
(forum) http://www.head-fi.org/
https://www.headphone.com/collections/top-10-headphones/audiophile
(srch results for headphones) http://tinyurl.com/mpt-headphones
Some words about Khyal
Perhaps the most popular of contemporary Hindustani vocal styles is the idiom known as "khyal." The word literally means "imagination," and the khyal form demands improvisational flexibility as well as careful attention to nuances of intonation, phrasing and rhythm. Paradoxically, it is less known to audiences in the West, who have learned to enjoy Indian instrumental music unhesitatingly -- and have even begun to absorb the intricacies of Dhrupad vocals. Music stores in India offer a wide selection of Khyal recordings, and performances are well attended.
Khyal is several hundred years old. Originating in the courts of the Moghul emperors as a less rigid alternative to the Dhrupad style, it has evolved into a remarkably flexible form that allows an artist's individuality considerable rein -- while remaining within the steady, inexorable flow of Indian tradition. Even within the past five decades the form has undergone metamorphoses, and the tradition of innovation continues. An artist is expected to develop an individual style (albeit one that is demonstrably connected to the tradition), and those performers who restrict themselves to mere imitation of their preceptors or of other famous musical personalities are firmly criticized for a lack of imagination.
Khyal texts draw freely from Hindu and Muslim poetic traditions, and are usually romantic or devotional -- or a combination of the two. Generally composed in the archaic Hindi dialect known as Brij Bhasha, khyal songs are also found in languages like Bhojpuri, Punjabi, Urdu, Rajasthani, Marathi and (occasionally) Sanskrit. Many songs have unclear wordings, and in some cases the texts have become completely garbled through multiple generations of oral transmission. However, the primary focus in a khyal performance is less on the textual or lyrical content of a song than on abstract musical values. Audiences respond with delight to nuances of ornamentation, to complex rhythmic improvisation, to intricate melodic patterns or to a vocalist's superb intonation.
Performances of khyal often start with a song in a very slow rhythmic cycle, perhaps of 10, 12, 14 or 16 beats; often the pulse is so slow that each beat is further subdivided -- for instance, a 12-beat cycle becoming one of 48. Each beat of the rhythm is marked by a specific stroke or combination of strokes on the tabla drums; by listening to their sound a singer can keep his or her place in improvisation. The tonal material of the raga gradually moves from a restricted melodic range to an extensive gamut; from a slow and relaxed pace to a quicker one. Improvisation is punctuated by the first few words of the song, which become a familiar melody leading up to the first beat of the rhythmic cycle. All extemporized melodic or rhythmic variations aim for this beat, and the gradual building of tension as it approaches is one of Hindustani music's great delights. Eventually the singer switches to a faster song in the same raga, displaying his or her virtuosity and command over the material. Some of the time a "tarana" is used to conclude -- these pieces are rhythmic in focus, and make use of nonsense syllables; they are thus in a sense analogous to scat singing in Jazz.
Fortunately, the music of great khyal singers (the proper generic noun is "khyaliya") is widely available on cassette, CD and lp. The following are some of the most highly regarded singers of this century; most are dead, but their music is easily obtainable, and their voices ring out with extraordinary conviction and passion. Other artists, like Bhimsen Joshi and Kishori Amonkar, are still performing and recording actively.
Kesarbai Kerkar
Hirabai Barodekar
Faiyaaz Khan
Bade Ghulam Ali Khan
Mallikarjan Mansur
Abdul Karim Khan
Amir Khan
Mogubai Kurdikar
Nazakat and Salamat Ali
Bhimsen Joshi
Kishori Amonkar
Contemporary performers include many musicians of great skill; some are regarded as "up-and-coming," while others are relatively senior performers who have only recently begun to attract critical attention. These include:
20 years of free stock data on Finnhub
[[legacy]]
blcked srch res: http://www.tinyurl.com/hvvf5lr http://www.ft.com/global-economy
http://www.economist.com/
https://www.bloomberg.com
http://www.mansionglobal.com/newyork
Rental studios under 2k
Studios for about $200/day - http://manhattan.marmaranyc.com/studio/ Brown harris (also has rentals) HDFC income-restr. apts on zillow. trulia has both for sale & for rent
(site below has WishList saved under email id)
http://www.furniturelandsouth.com/
http://www.hickoryfurniture.com/Furniture-Jonathan-Charles-Tables/ItemBrowser.aspx?action=attributes&ItemType=Furniture&Brand=Jonathan%20Charles&Type=Tables&TrailIndex=1
http://s3.amazonaws.com/images2.eprevue.net/p4dbimg/513/images/217%20jessicamcclintock%20cover.pdf
Leather- Barrel-Chair
Miles Talbott Living Room Chair
Tufted Leather Chair by Reflections
Lexington Living Room Cardiff Chair
Thomasville Vienna Chair
Lincoln- Tufted-Chair
Kenneth Chair
George III Wing Chair
George III Tufted Wing Chair
Chandler Chair
Michael Amini Knightsbridge Charterhse Leather Chair
Michael Amini Chair and a Half
Francesco Molon Leather Wing Chair
Small Marcelle Chest
Floral painted chest
Chaddock- Petite-Camille-Small-Chest
Chatelet Floor Mirror
Furniture Rhapsody Floor
Black Gold Drinks Cabinet
Black Chinoiserie Drinks Cabinet
Chaddock- Prince-Bar-TV-Cabinet
Guildmaster Provence Drink Cabinet
Mahogany Drum Table
Walnut Eglomise Side Table
Rope Twist Round side table
Lapis Blue Eglomise With Mahogan
Jonathan Charles Tables
Artitalia Group Armoire
Francesco Molon 2 dr armoire
Artitalia Group Credenza
Artitalia Group Credenza in Oak
Hooker furniture vintage king poster bed
Guildmaster Heritage Caned Bench
Note: This design can also be used for a bench swing (custom)
Karges Furniture Georgian Chest
Four Hands Large desk
Hooker Furniture Kendrick jr exec desk
Hooker Furniture la maison Writing desk
Hooker Furniture Sanctuary 60" Writing Desk
Lorts Three Drawer Writing Desk
Theodore Alexander Chippendale's Signature Writing Desk
Manhattan Mini Storage www.manhattanministorage.com
Makespace https://makespace.com/nyc/
Box Butler http://www.yelp.com/biz/box-butler- new-york
OR dump stuff in Edison/Philly (cheaper + less crowded + less hassles)
NOTE: MOST MANHATTAN SITES DON'T HAVE TRUE 24X7 ACCESS, ONLY DAYTIME ACCESS. ALSO, IF THEY DELIVER THERE MIGHT BE DELAYS etc.
JAN. 22, 2013 Gotham Mini Storage
Prices range from $39 a month for a 64- cubic-foot locker reached by a stepladder, to $2,000 for a garage- size space. In contrast, a similar 64-cubic-foot space at Manhattan Mini Storage this week was $57 a month, with a year lease, at most locations. Prices started at $70 a month, for a slightly larger space, at Tuck-It-Away, which is based in Upper Manhattan.
2016:Gotham Mini Storage Locker Promotion Rent a storage locker for $79/month at Gotham Mini Storage and save 50% for the first three months.
There are about 30 self-storage facilities in Manhattan. Manhattan Mini Storage, which has been in business since the 1970s, owns 17 of those properties. (NYT 08:) A 10-by-10-by-8-foot space runs about $300 a month. manhattanministorage.com. Spaces start at 4 by 4 by 5 feet, and one of the most popular is a closet that size with a hanging rod and shelves, "so people can store out-of-season clothing,"" he said. The monthly cost is $45. The company provides a key card for access to the building and elevator, and the customer provides a lock.
CubeSmart ... enjoy a range of new services, like a courier that can shuttle boxes to tenants' homes its employees can sign for packages, which is important for tenants who run their businesses out of their storage areas and who make up about 30 percent of CubeSmart's tenants
Tuck-It-Away, one of the oldest operators in the city, owns 14 facilities with a total of about one million square feet of space.
''Most people rent twice as much space as they really need,'' said Edward Troianello, the president of the Self-Service Storage Association of New York. So calculate carefully and conservatively. According to Mr, Troianello, a three-piece bedroom set, plus the boxed contents of a bedroom closet, can fit into a 5-by-5-by-8-foot storage room
(NYT 14:)start-ups like MakeSpace charge roughly $220 a month for 36 square feet of space at an offsite warehouse.
https://www.publicstorage.com/storage-search- landing.aspx?location=edison+nj
Storage Units Off Old Post Road in Edison, NJ --
SMALL 5' x 5'
*Climate Controlled
*Inside unit/1st Floor 1 Left $69/mo.
Online Special* $74 In-store
(Note: These work in lib browser)
Some girl who moved from Minn to NYC - http://101baddesidates.blogspot.com/
Amardeep Singh (dated site but lotsa links) = http://www.lehigh.edu/~amsp/2005/04/desi-weekend-indian-bloggers-bob- nyc.html
SruTHI's decorating website - http://theeastcoastdesi.blogspot.com/
SAJA Form desi spotting - http://www.sajaforum.org/desi_spotting/ Professional Indian Associations (net-ip, tie, etc.) Valley desi site
Bhel, chaat, etc @ Murrah Hill & Village http://www.desi-galli.com/
Nisha's Desi Street Food Site LOTS of info - http://indianstreetfoodie.com/
punjabi forum (chk out jokes)- http://www.desicomments.com/desi/janmashtami/
Daandiyaa TOO! host events all over, website only works in IE -> www.desiparty.com
Neill Rea (Actor)
Filmed amid the beautiful landscape of New Zealand's North Island,
these four feature-length mysteries show that murder can haunt even the most idyllic locations.
from an AMZN review:
The last three years have taken a dreadful toll on my favorite British policemen:
"Foyle's War" ended in 2015 after eight seasons.
"New Tricks" ended in 2015 after twelve seasons.
"Inspector Lewis" ended in 2016 after nine seasons.
"DCI Banks" ended in 2017 after five seasons.
"George Gently" will end in 2017 after eight seasons.
(we may come to look back on the last ten years as the Golden Age of British Police on Television.)
The one bright spot is "Endeavour".
This prequel to "Inspector Morse" is still going strong in it's fourth season.
A fifth season has been announced for 2018.
"Endeavour" debuted in 2013, but was set in 1965, when Constable Endeavour Morse joined the Oxford City Police.
Promoted to Sergeant Morse in 1969.
If it's still on the air ten years from now, we will see the return of "Inspector Morse".
If it's still on the air eighteen years from now, we will see the return of Sergeant Robbie Lewis.
WHY YOU SHOULD WATCH ENDEAVOUR ON BLU-RAY OR DVD, NOT PBS:
I always knew Masterpiece Mystery broadcasts on PBS were edited, but was I was surprised to learn exactly how much was cut.
Wallander
Kenneth Branagh returns to his remarkable Emmy and Golden Globe-nominated role
as the soul-searching Swedish cop in Wallander III, with three gripping new Kurt Wallander
cases based on the character created by bestselling novelist and father of the
Nordic noir craze, Henning Mankell.
Worricker
2013
Page Eight, Turks & Caicos, and Salting the Battlefield form The Worricker Trilogy
three gripping films that follow the exploits of Worricker
from MI5 headquarters in London to exile on a Caribbean island to life on the run
with his former lover and fellow agent Margot Tyrrell.
The all-star cast includes Oscar winner Christopher Walken.
Arthur & George
2015
Martin Clunes (Doc Martin) stars as Sir Arthur Conan Doyle, the creator of
Sherlock Holmes, in a real-life case that inspired the great author to put
down his pen and turn detective.
Co-starring Art Malik
Inspired by a true case, Arthur & George is adapted from Julian Barnes's
acclaimed novel of the same name
List of Masterpiece Mystery! episodes
List of Masterpiece Contemporary episodes
List of Masterpiece Theatre episodes
Season 32 (2002-03)
Almost a Woman
Daniel Deronda
The Forsyte Saga
Foyle's War
The Hound of the Baskervilles
The Jury
Me & Mrs. Jones
My Uncle Silas II
White Teeth
List of Masterpiece Classic episodes
(2008 onwards)
Originated in Eastern Uttar Pradesh, mainly in Lucknow & Benares, around the 18th century.
A romantic & erotic style of singing; also called “the lyric of Indian classical music”.
Compositions are mostly on love, separation, and devotion.
Distinct feature: Erotic subject matter portrayed picturesquely from the various episodes of the lives of Lord Krishna & Radha.
Lyrics are typically in Brij Bhasha and are usually romantic & religious.
A Thumri is usually performed as the last item of a Khayal concert.
Three main gharanas of thumri — Benaras, Lucknow, and Patiala.
Begum Akhtar is one of the most popular singers of the thumri style.
Kathak uses ‘Thumris’ (expression-based numbers like ‘Padams’ and ‘Javalis’ of Bharatanatyam) to tell stories – stories of love, anger, separation, heroes and heroines. Thumri originated from the term ‘Thumakna’ which refers to rhythmic movement and stylized gait. These semi-classical compositions combined melody and rhythm with a couple of lines of text/ poetry, which was more colloquial. Some Thumris reflected Shringara rasa (the amorous flavour of love and romance) and some, ‘Bhakti’ (devotion).
While the early Kathakars and forest dwellers told the stories of the divine (early Gupta period 4th century CE), the traditional compositions also explored love and beauty with Shyam/ Krishna as the hero of the story....
...ater, rulers like Nawab Wajid Ali Shah went on to write Thumris, set them to music, and also danced to various Thumris, where he played the role of Krishna. The others like Pia Rangeelay, explored the aspect of love through Thumris, where the subjects were the lover and his beloved, and not specifically Gods as heroes of the Thumris, as it used to be in the traditional compositions. Much later, during the British rule, when Indian classical dance was considered nautch, the presentation of the Nayika became more sensuous and enticing, the style of singing more provocative, and thus a lighter version of the Thumris emerged.
(from namasteswitzerland)
(from a film review: Girija- A Life Time in Music)
Produced by Madhu Chandra and Sudha Datta and directed by Debapriya Adhikary, Samanwaya Sarkar and Sankalp Meshram; with a running time of over 2 hours, Girija- A Life time in Music is a labour of love that takes us into the world of Padma Bhushan , Girija Devi- one of the finest contemporary Hindustani vocalists of our times.
In the film, a student of Girija Devi recalls an incident- of how one day Appa ji, as Girija Devi is lovingly referred to by her students and younger members of the music fraternity, was sitting in the verandah and suddenly called out to her. It was drizzling outside, she recalls and Appaji asked her to extend her arm and feel the rain drops. “Feel it,” she told her-“this is the jhir jhir barse sawan ras bundiya that we sing about. Feel how softly the drops fall. Now if you take a big taan while singing of rain that would mean musladhaar baarish (heavy rain), so the taans need to be soft and delicate when you speak about jhir jhir rains.”
This one scene gave me goosebumps. As did many other… The documentary is such a wonderful attempt to introduce music lovers to the living legend (who...) rose to become the epitome of grace in the world of Hindustani music and earned the title of Queen of Thumris.
(below snipped from good short essay here)
The Banarasi style draws from poetry in Hindi, Braj and Avadhi to source suitable insertions, while Lucknowi thumri sources insertions from Urdu poetry alone. While Lucknowi thumri shows marked influence of Kathak, ghazal and tappa, the Banarasi variety seems more inclined towards adapting melodies from the folk repertoire of kajri, chaiti and jhoomar. Lucknowi thumri illustrates the impact of the courtly culture of the Nawabs of Avadh, while Banarasi thumri uses with elements from folk culture...
Some of the most famous thumri artists are – Badi Motibai, Rasoolan Bai, Siddheshwari Devi, Savita Devi, Girija Devi, Gauhar Jan, Shobha Gurtu, Begum Akhtar, Prabha Atre, Pandit Channulal Mishra, Naina Devi, Purnima Choudhuri, Shubha Mudgal, Abdul Karim Khan, Nazakat-Salamat Ali Khan, Barkat Ali Khan, Bade Ghulam Ali Khan. Among these stars Shobha Gurtu is the brightest and is called “the thumri queen”.
https://www.quora.com/What-are-some-of-the-best-thumris
has links to YouTube incl SAIGAL Babul Mora
The Hori Thumris are most often based on the romance of Krishna and Radha. They talk about their play and pranks with colors during Holi.
an excellent, in-depth essay on Hori Thumris w/links to many YouTube entries
All Episodes of In our Time
Episodes classified by the Dewey Dec. Sys
A few selections (only chkd upto about #500)
Billboard #1 hits over the years
Grammy-nominated or not, No. 1 hits over the years (APnews: By MESFIN FEKADU) NEW YORK (AP) The exclusion of The Weeknds Blinding Lights at the 2021 Grammy Awards shocked many, but hes in good company: Princes When Doves Cry never scored a nomination either. Heres a look at every Billboard No. 1 hit of the year since 1958, Grammy-nominated or not. NOTE: Songs with an asterisk represent tracks that earned a Grammy nomination; songs with two asterisks won a Grammy. 2020: The Weeknd, Blinding Lights 2019: Lil Nas X featuring Billy Ray Cyrus, Old Town Road (asterisk)(asterisk) 2018: Drake, Gods Plan (asterisk)(asterisk) 2017: Ed Sheeran, Shape of You (asterisk)(asterisk) 2016: Justin Bieber, Love Yourself (asterisk) 2015: Mark Ronson featuring Bruno Mars, Uptown Funk (asterisk)(asterisk) 2014: Pharrell Williams, Happy (asterisk)(asterisk) 2013: Macklemore & Ryan Lewis featuring Wanz, Thrift Shop (asterisk)(asterisk) 2012: Gotye featuring Kimbra, Somebody That I Used to Know (asterisk)(asterisk) 2011: Adele, Rolling In the Deep (asterisk)(asterisk) 2010: Kesha, Tik Tok 2009: Black Eyed Peas, Boom Boom Pow (asterisk)(asterisk) 2008: Flo Rida featuring T-Pain, Get Low (asterisk) 2007: Beyonc, Irreplaceable (asterisk) 2006: Daniel Powter, Bad Day (asterisk) 2005: Mariah Carey, We Belong Together (asterisk)(asterisk) 2004: Usher featuring Lil Jon and Ludacris, Yeah! (asterisk)(asterisk) 2003: 50 Cent, In Da Club (asterisk) 2002: Nickelback, How You Remind Me (asterisk) 2001: Lifehouse, Hanging by a Moment 2000: Faith Hill, Breathe (asterisk)(asterisk) 1999: Cher, Believe (asterisk)(asterisk) 1998: Next, Too Close 1997: Elton John Candle In the Wind 1997 (asterisk)(asterisk) 1996: Los del Ro, Macarena (Bayside Boys Mix) 1995: Coolio, Gangstas Paradise (asterisk)(asterisk) 1994: Ace of Base, The Sign (asterisk) 1993: Whitney Houston, I Will Always Love You(asterisk)(asterisk) 1992: Boyz II Men, End of the Road (asterisk)(asterisk) 1991: Bryan Adams, (Everything I Do) I Do It for You (asterisk)(asterisk) 1990: Wilson Phillips, Hold On (asterisk) 1989: Chicago, Look Away 1988: George Michael, Faith 1987: The Bangles, Walk Like an Egyptian 1986: Dionne Warwick & Friends, Thats What Friends Are For (asterisk)(asterisk) 1985: Wham!, Careless Whisper 1984: Prince, When Doves Cry 1983: The Police, Every Breath You Take (asterisk)(asterisk) 1982: Olivia Newton-John, Physical (asterisk) 1981: Kim Carnes, Bette Davis Eyes (asterisk)(asterisk) 1980: Blondie, Call Me (asterisk) 1979: The Knack, My Sharona (asterisk) 1978: Andy Gibb, Shadow Dancing 1977: Rod Stewart, Tonights the Night (Gonna Be Alright) 1976: Wings, Silly Love Songs 1975: Captain & Tennille, Love Will Keep Us Together (asterisk)(asterisk) 1974: Barbra Streisand, The Way We Were (asterisk)(asterisk) 1973: Tony Orlando and Dawn, Tie a Yellow Ribbon Round the Ole Oak Tree (asterisk) 1972: Roberta Flack, The First Time Ever I Saw Your Face (asterisk)(asterisk) 1971: Three Dog Night, Joy to the World (asterisk) 1970: Simon & Garfunkel, Bridge Over Troubled Water (asterisk)(asterisk) 1969: The Archies, Sugar, Sugar 1968: The Beatles, Hey Jude (asterisk) 1967: Lulu, To Sir with Love 1966: SSgt. Barry Sadler, Ballad of the Green Berets 1965: Sam the Sham & the Pharaohs, Wooly Bully (asterisk) 1964: The Beatles, I Want to Hold Your Hand (asterisk) 1963: Jimmy Gilmer and the Fireballs, Sugar Shack 1962: Acker Bilk, Stranger on the Shore (asterisk) 1961: Bobby Lewis, Tossin and Turnin 1960: Percy Faith, Theme from A Summer Place (asterisk)(asterisk) 1959: Johnny Horton, The Battle of New Orleans (asterisk)(asterisk) 1958: Domenico Modugno, Nel Blu Dipinto di Blu (Volare) (asterisk)(asterisk)
Stephen Holden writes for the NYT covering the Cabaret/Standards Scene (Gershwin/Berlin/Porter/Broadway)
Show Calendars @ some NYC venues
http://metropolitanroom.com/enhancedCalendar.cfm https://54below.com/calendar/?month=December+2016 https://www.rosewoodhotels.com/en/the-carlyle-new-york/location/things-to-do/events-at-the-carlyle http://www.birdlandjazz.com/calendar/ http://www.jazz.org/calendar-full.php http://americansongbook.org/ "NYC's 3 major cabaret supper clubs - Cafe Carlyle, the Oak Room (Algonquin) & Feinstein's @ Loews Regency - are attached to hotels. Other downtown Cabaret venues listed in a NYT "Cabaret for a steal" article by Erik Piepenburg: the 70-seat cabaret theater @ the Duplex in the West Village cover is $20 w 2 drink min. 100-seat place @ Laurie Beechman Theater in Clinton Don't Tell Mama, W 46th St. the Metropolitan Rm Le Poisson RougeJoe's PubCast Party @ Birdland Kazuo Ishiguro "writes fanciful, mildly surreal lyrics to (Jim) Tomlinson's (saxophonist) music..." ...latter-day answer to Gilberto & Stan Getz (!!)
Harmonica stuff
Harmonica Notes: Understanding the holes for playing songs
If you play any blow hole on a standard key of C 10-hole diatonic harmonica will the pitch be higher or lower?
If you dont know the answer to this trick question, you may have trouble playing songs forever.
Lets figure it out together.
First, blow into hole number 1 and follow that by drawing into hole number 1. Which is higher? The draw note is higher. Right? Check for yourself and youll see this is correct.
Now blow into hole number two, then draw. Again the draw note is higher.
So from these two examples you might conclude that the draw notes are higher than the blow notes, but, if so, you would only be partly correct.
Now try blowing into 10 and then drawing into 10. Hopefully your ear can tell you this is different. Which note is higher in pitch?
The blow note is the higher pitch than the draw note. Somewhere on the harp, the pattern goes from the draw notes being higher to the blow notes being higher. Where does that happen?
Blow draw into 3, then 4, then 5, then 6, then 7! The switch happens between holes 6 and 7. From holes 1 through 6 the blow notes are lower than the draw notes. From holes 7 through 10, it reverses. The blow notes are higher than the draw notes.
Because of this, most harmonica players call holes 1 through 6 the low end of the harp and holes 7 through 10 the high end of the harp. Many players get confused when exploring the high notes.
They know something is different, but they cannot put it into words. Because of this, many harmonica players do not continue exploring the high notes (and they feel intimidated to play songs on the higher notes).
Have the courage to explore the high notes and, the more you do it, the easier it will be for you to include all the notes on the harmonica to play the songs you love.
Playing the major scale is a great way to feel the shift.
Play these notes to go up the major scale:
Do Re Mi Fa So La Ti Do
4 -4 5 -5 6 -6 -7 7
And play the same notes in reverse to go down the major scale.
Practice this a bit and this initially confusing aspect of harmonica will soon become second nature to you.
(Article by: Michael Rubin)
Wild Thing (Hendrix)
5 4 -5 -5 5 -4 4
Wild Thing you make my heart sing
-5 -5 6 4 4 4 3
You make everything groovy
-4 4
Wild Thing (spoken)"Wild thing I think I love you"
srch results for manhattan veggie places: http://tinyurl.com/z9bh8qq
- 252 Manhattan veggie restaurants at http://www.vegguide.org/region/5
(but v. few rated well, lots of cafes & raw/vegan stuff) top choices have been listed below.
Hummus Kitchen - 768 9th Avenue 212-333- 3009
Gd rev from NYT, also has falafel
Soy and Sake Village
47 7th Avenue South 212-255-2848 http://www.soyandsake.com
Japanese Vegetarian cafe West Village specializing in Japanese food, but also carrying Chinese, Malaysian, and American soul food. owned by the same people as Red Bamboo / VP2, so the menu includes many well-known Red Bamboo items like chicken Parmesan, soy chops, and other soul food dishes utilizing soy meats. Other menu items include: numerous mockmeat sandwiches, curries, salads, noodle dishes, bento boxes and sushi. Lunch specials are about $7-9 include both sushi and soul food options.
Hangawi 12 E 32nd Street
Vegan Korean dining in a tranquil space "posh". Zagat rated as top Korean and top vegetarian.
Lunch specials are WAY cheaper than dinner ($10 vs. $25),
Beyond Sushi 229 E 14th St betw 2nd & 3rd Ave Union Square vegetarian sushi shop
Executive Chef Guy Vaknin is a Hell's Kitchen finalist. He serves all vegetarian sushi made from fruits and vegetables topped with customized sauces. Spicy Mango Roll - black rice, avocado, mango, cucumber and spicy veggie topping with toasted cayenne sauce.
Kyotofu Upper West Side 705 Ninth Avenue
Upscale, modern Japanese food with an emphasis on tofu
Souen Organic Ramen East Village Betw 1st & 2nd Ave 326 E 6th St Small organic ramen noodle shop
From the owners of Souen comes this tiny organic ramen noodle shop. The seitan dumplings are vegan but make sure to ask questions about the ingredients of the different broths available.
Maoz Vegetarian Restaurant 38 Union Sq E International Falafel Chain
falafel sandwiches, salads, and fries. After you get your sandwich, there's a large selection of self-serve toppings - pickles, cucumber salad, roasted cauliflower, dairy and vegan coleslaws, carrots, baby eggplants dyed in beet juice (really), olives, several fresh hot sauces, and much more. The trick is they don't leave very much spare room in your sandwich, so you can't really load up on the condiments. The first Maoz branch was opened in 1991 in Amsterdam, Holland
KAJITSU it's Michelin-starred Midtown E 125 E39th St betw Park & Lex Aves
A tasting menu here runs $95, with the option of an additional $59 sake pairing;
you can also opt for the less extravagant $55 menu, with a $45 sake pairing.
Shojin cuisine refers to a type of vegetarian cooking that originates in Zen Buddhism.
Even though it does not use meat or fish, shojin is regarded as the foundation of all Japanese cuisine, especially kaiseki, the Japanese version of haute cuisine. In its present form kaiseki is a multi-course meal in which fresh, seasonal ingredients are prepared in ways that enhance the flavor of each component, with the finished dishes beautifully arranged on plates. All of these characteristics come from shojin cuisine, which is still prepared in Buddhist temples throughout Japan.
Peacefood Cafe 460 Amsterdam Ave betw82nd and 83rd Streets on UWside
The roasted Japanese pumpkin sandwiches ($9.75) are served on spelt bread and come stuffed with pumpkin pulp, cashew cheese, walnuts and onions; the Shangai-style dumplings ($8.50) are packed with chives, mushrooms, marinated tofu and vegetarian protein, and the chickpea fries ($8.50) are thick mashed legume slices fired up with piquant Indian spicethere are also vegan cheeseburgers on tap at Peacefood's downtown spot. Don't skip out on dessert here, since the raw key lime pie ($7.50) will blow your mind, almond brazil nut crust and all. (limes shortage coming up).
In addition to expanding service to anyone and everyone (in Midtown East) Jul2016, David Chang's delivery-only Momofuku restaurant Ando also introduced a new vegetarian sandwich and a meat-less side: Veg Banh Mi s/w ($10), Kimchi salad $4
Charrua 131 Essex Street, New York, NY 10002 (212) 677-5838 (S of Houston)
the vegetarian chivito, called the el cheto, a "signature Uruguayan sandwich" filled with portobello mushrooms and mozzarella, plus carrots, caramelized onions, and chimichurri, and served with a handful of thick, hot fries.
Renovations are now complete at Tribeca stalwart Pakistan Tea House. The inexpensive restaurant was purchased by the owners of Baluchi's last year, who decided to close the shop for some renovations in January. The space now has a few new details including large wooden tables, and chrome panelling on part of the wall but the overall layout remains the same. Tribeca Citizen notes that the restaurant is now serving a limited menu, which includes vegetarian and meat combo plates for $8 to $10. According to the specials board, samosas are two-for-a-dollar.
Cinnamon Snail at The Pennsy
World famous vegan food truck turned permanent vendor in this spacious food hall above Penn Station. Every vegan's dream
Founded by renowned Chef, Guy Vaknin, and his wife, Tali, Beyond Sushi offers an assortment of distinctive sushi rolls and other nutritional staples at their three outposts in Manhattan.
Chef Guy and his culinary team fuses unconventional pairings of fruit and vegetables with whole grains to create sushi rolls. Hand pressed juices; reason. prices, online ordering/deliv.
UNION SQUARE 646-861-2889 229 E. 14th St. (Between 2nd & 3rd Ave)
MIDTOWN WEST 646-964-5097 62 W 56th St. (Between 5th & 6th Ave)
CHELSEA MARKET 212-929-2889 75 9th Avenue (Between 15th & 16th St)
Cinnamon Snail Penn Plaza, Southwest corner of 33rd St. and 7th Ave
"voted the top food truck in America", four- time Vendy Award winner the Cinnamon Snail finally shares the recipes for its beloved street-side meals. Here you'll find Fresh Fig Pancakes for breakfast, Chimichurri Tempeh Empanadas for a snack, Thai Barbecue Seitan Ribs for a meal, and, of course, Snail classics like Vanilla-Bourbon Creme Brulee Donuts for dessert.
Blossom du Jour (212) 229-2595 Vegan, Juice Bars & Smoothies 5 locations:
CHELSEA 259 West 23rd Street (between 7th & 8th Ave.)
HELL'S KITCHEN 617 9th Avenue (between 43rd & 44th St.)
UNION SQUARE 15 East 13th Street (between 5th & University)
UWS 81ST 449 Amsterdam Avenue (between 81st & 82nd St.)
COLUMBUS CIRCLE in the subway statione 1000S 8th Ave, #21
For over a decade, Terry's Gourmet Deli on Chelsea's 6th Avenue has dispensed some of the city's best Trinidadian rotis. The vegetarian (potatoes and chickpeas only) weighed in at a whopping 1 pound, 10 ounces; was 6 inches long, and cost $8. I'm not sure how one person could finish it. You can ask the roti maker for any amount of pepper sauce to be put inside, and I usually go for three plastic teaspoonfuls. 575 6th Ave, (212) 206-0170.
Nix (Michelin-starred chef John Fraser)
Vegetables are finally getting their due in New York, as evidenced by new all-vegetarian restaurant Nix. The food is complemented by a strong drinks program at this NYU-area restaurant in a modern space.
(Brdway & 11th, just north of the Village) 72 University Pl New York, NY 10003 (212) 498-9393
presently open for dinner only. http://www.nixny.com/
"When my vegetarian friends ask for dining advice I send them to American spots like Hearth, for hen of the woods mushrooms and gnocchi, or to to nouveau-Indian spots like Babu Ji for their bright yellow Punjabi curries. I send them to creative pizza spots like Roberta's and Pasquale Jones, to Semilla for vegetable-heavy tastings, to Del Posto for their vegan tasting menu. Some of the most celebrated vegetable dishes right now are served at "regular" restaurants: the grain bowls, the avocado toasts, the brown butter and squash carpaccios.
Superiority Burger made the veggie burger just as much an object of desire among the city's culinary cognoscenti as a trendy lamb burger. Now, the menus at Momofuku restaurants read "vegetarian options available on request."
famed Avenue B deli Sunny and Annie's - vegetarian sandwich - had not just pesto, grilled mozzarella, fresh goat cheese, spinach, seaweed, and avocado, but also ~three~ pieces of bread, making it the first non-meat double decker I've ever seen.
new fast-casual Asian restaurant @ 122 Ludlow Street Zing's Awesome Rice: menu will include six types of "seared rice," Yes, the fried rice is awesome, and it comes attractively plated with two domed scoops on either end of a long plate, with sriracha squirted in the middle in the shape of a heart or a leaf. But is it worth $11.99 to $13.99 for a modest serving, so that a full meal with app and beverage will run you $20 or $25? That's for you to decide. The rice is loaded with umami, has diced vegetables in it, and offers as a main ingredient a choice of tofu. For an extra dollar or two you can pick a more exotic form of rice, including a black rice with a particularly nutty flavor. Beverages run to beer and infused sake.
(5/16) Philadelphia-based chef Michael Solmonov and restaurateur Steve Cook officially kicked open the doors to the Chelsea Market outpost of their hit hummus restaurant, Dizengoff, this morning. A key member of the Dizengoff Philly team moved to New York for the opening of this new counter: executive chef Emily Seaman. Dizengoff will always serve fresh hummus, and pita baked in a taboon, but the toppings and combinations will change often. Solomonov tells Eater: "Emily really changes the menu all the time. She gets everything from the market, basically, and transforms it into Israeli-esque dishes." Solomonov notes that the avocado with peanut harissa on the opening menu is "not something that you'd ever find in Israel, but it really tastes fantastic." The hummus meals are $10 to $13, and the salatim are $3 each. for the moment, Dizengoff NYC is serving salads, hummus, pita, and drinks.
<<resp to above, a reviewer sez:>>I thought that the hummus at Dizengoff was excellent - probably the best in Manhattan along with Baraca on 38th Street, and amongst the best in the entire City in league with places like Olympic Pita and Mabat in Brooklyn, or Grill Point in Queens. (they don't have falafel for some reason)
Freud
The brunch menu at chef Eduard Frauneder's new Greenwich Village Austrian restaurant has a vegetarian breakfast sandwich with cauliflower, egg, Tumbleweed cheese, and onions on a roll. Baked eggs are also on the menu. For sweet options, Frauneder is offering pancakes with rhubarb, and French toast served with walnuts and smoked maple syrup.
506 Laguardia Pl New York, NY 10012 (212) 777- 0327 (just N of Houston below the Village)
Stuff to eat @ the Columbus Circle's Turnstyle Food Court (access via an outdoor elevator at 59th Street and 8th Ave)
2. Tex-Mex Slice at The Pizza * Uber-cheesy, with a slightly thicker than usual crust dotted with serious jalapenos and black-olive slices, and irrationally finished with squiggle of honey, the Tex-Mex slice verges on wonderful. But they should hand out wet-naps due to stickiness. $3.50
3. Key Lime Tart at Georgia & Aliou's * Topped with a fresh raspberry and a few slivers of lemon zest, the yellow pastry looks like a beach scene if you squint your eyes. It's every bit as tart and rich as you hoped, made for nibbling rather than biting. $5.50
5. Smoked Egg Salad Sandwich at Ellary's Greens * Vegetarians take note: The smoked egg salad sandwich at Ellary's Greens delivers much the same barbecue power as a slice of smoked brisket, with chopped kale adding to the textural terrain. $8
6. Avellino Focaccia Sandwich at Casa Toscana * The house-baked focaccia (sometimes known as pizza bianca) is thin and slightly crisp, and on it is piled mozzarella, cherry tomatoes, and fresh basil, making for a simple and delicious sandwich, the way they do it in Italy. $9
Agern
Agern is the new restaurant from Noma co- founder Claus Meyer and chef Gunnar Gaslason. It is located within Grand Central Station, right next to the Nordic food court that Meyer recently opened. The focus is primarily on Nordic cooking using local American ingredients. There are two tasting menus available * the vegetarian "Field and Forest" ($120) and the meat and fish heavy "Land and Sea" ($145) * as well as a la carte options. Both tasting menus span seven courses, along with a selection of amuse-bouches. (Prices are inclusive of service.)
89 E 42nd St New York, NY 10017 (646) 568- 4018
Salvation Burger - rustic joint in the Pod 51 hotel
Chefs across New York are fostering a vegetarian burger renaissance, and the version April Bloomfield serves at Salvation Burger, her newest restaurant, ranks near the top of the citywide hierarchy. Her kitchen mashes together golden beets, carrots, carrot juice, French lentils, mushrooms, sticky rice, and sweet potato vermicelli. This is all laced with enough warming garam masala to power a geothermal turbine, grilled over wood, and then anointed with cooling yogurt, shredded lettuce, and sweet tomato confit.<<writer goes on to pan this place bigtime, also v. exp (~30 burger) Desi spots:
Founded in 1975 with a name certain to popularize the cuisine to a wider audience, Curry in a Hurry was one of the original anchors of Curry Hill. The international food store Kalustyan's was another. 119 Lexington Ave, (212) 683-0900.
Dating to 1996, the amazing Vatan still offers a single enormous vegetarian meal, reflecting an era when the westernmost state of Gujarat provided many of the city's Indian immigrants. The price of the meal is fixed at $32, and includes approximately 25 dishes organized into four courses. It's also all-you-can-eat. 409 3rd Ave, (212) 689-5666.
<<just a reg. thaali, avoid>>
Bombay Masala (148 W 49th St #2, 212-302-8150) this Times Square standard, successor to Ceylon India Inn, which occupied the space as late as 1968, has a $12.95 lunch buffet considered a good deal.
Vegetarian Ham Banh Mi * Vietnamese baguette sandwiches present such a chorus of powerful flavors (pickled veggies, cilantro, jalapenos, doctored mayo) that it's hard to imagine that lack of real meat would make much of a difference. But there's something wonderful about the fake ham used in this banh mi * greasy, salty, and gelatinous that actually boosts the flavor of this excellent sandwich. Find it at: Banh Mi Place, 824 Washington Ave, Brooklyn,
Tuning for Lower Response Time
Use the following commands and advanced runtime options to achieve lower response times for your application:
java -XX:+UseG1GC -Xms26g Xmx26g -XX:MaxGCPauseMillis=500 -XX:+PrintGCTimeStamp
(for 32bit max is 2G; the foll works for WF->
java -XX:+UseG1GC -Xms1200m -XX:MaxGCPauseMillis=500 -XX:+DisableAttachMechanism -cp lib\c.jar;lib\fatty.jar;bin com.trivedi.launcher)
Configuring the JVM, Java Options, and Database Cache
https://blog.gceasy.io/2020/03/18/7-jvm-arguments-of-highly-effective-applications/
(includes mem settings XMS + choice of GC, etc.)
6.-Dsun.net.client.defaultConnectTimeout and -Dsun.net.client.defaultReadTimeout
sun.net.client.defaultReadTimeout specifies the timeout (in milliseconds) when reading from the input stream when a connection is established to a resource.
Example, if you would like to set these properties to 2 seconds:
-Dsun.net.client.defaultConnectTimeout=2000
-Dsun.net.client.defaultReadTimeout=2000
Note, by default values for these 2 properties is -1, which means no timeout is set
Instantiation suppliers
Suppliers are useful for lazy initialization, and whenever a Supplier is expected, you can pass Type::new.
A constructor reference E::new is in fact a Supplier. That means wherever you have suppliers you can use it. If you are using the stream API, you will inevitably find some use-cases, e.g. Stream#toArray(String[]::new) or Collectors#groupingBy() (or in general, collectors) to instantiate a collection or map of a specific type instead of relying on the type returned by default.
AtomicReference
The compareAndSet method takes two parameters, the expected current value and the new value. The method atomically checks if the current value equals the expected value. If yes, then the method updates the value to the new value and returns true. If not, the method leaves the current value unchanged and returns false.
We atomically check if the current value is still the same as when we started the computation.
If yes, we can safely update the current value. Otherwise, we need to recalculate the new value with the changed current value.
A Type-safe actor model for java A Type-safe actor model for java
Akka Typed Actors use JDK proxying to wrap an object and return a safe interface that converts normal Java method calls to asynchronous.
Jumi Actors have a similar approach
This article mentions how to do the same thing by wrapping the method call in a high-order function rather than proxying an interface
Dynamic Attach
The Java API for Dynamic Attach is available in the same tools.jar file. Note that this is a vendor-specific API applicable only to OpenJDK and Oracle's JDK.
Attaching to a running Java process is straightforward; you need to know only the target process ID (pid) as shown in the following code. (Dynamic Attach requires no special VM options. It can connect to any local HotSpot JVM unless it is started with the -XX:+DisableAttachMechanism flag.)
HotSpot Serviceability Agent
The HotSpot Serviceability Agent (SA) provides a low-level view of a Java process from a VM perspective.
The SA provides the reflection-like API to inspect Java objects and to extract the required fields. Unlike the reflection, which works from within the same process, the SA reads memory of a different process or parses a core dump file. Tools based on this feature can do impressive tricks such as stealing private keys from a running web server.
The SA needs no cooperation from the target JVM, and there is no way to protect against SA interactions. This is not a reason to worry, though. The SA typically requires root privileges to attach to a running process. Also, keep in mind that the target JVM remains suspended while the SA is attached.
From javaMag / TextPad ->
// Don't close automatically, WindowCloser will exit when done.
setDefaultCloseOperation(WindowConstants.DO_NOTHING_ON_CLOSE);
this.addWindowListener(new WindowAdapter() {
public void windowClosing(WindowEvent e) {
doWhenNoUnsavedChanges(() -> {
setVisible(false);
dispose();
System.exit(0); });
}
});
/**
* Check for unsaved changes; if so, prompt.
* When approved, perform the action passed in.
*/
public void doWhenNoUnsavedChanges(Runnable r) {
if (mUnsavedChanges) {
int ret = JOptionPane.showOptionDialog(TinyPad.this,
"You have unsaved Changes!", "Warning",
JOptionPane.YES_NO_CANCEL_OPTION, JOptionPane.WARNING_MESSAGE,
null, unsavedOptions, unsavedOptions[0]);
switch(ret) {
case UNSAVED_OPTION_SAVE: // Save
saveActionListener.actionPerformed(null);
if (mUnsavedChanges) { // ie., save failed
return;
}
break;
case UNSAVED_OPTION_DISCARD: // Discard
break;
// NOTREACHED
case JOptionPane.CANCEL_OPTION:
case JOptionPane.CLOSED_OPTION:
// Do nothing; in particular, do not exit.
return;
default:
throw new IllegalStateException("Unexpected: " + ret);
}
}
// We got here, so run the Runnable.
r.run();
}
WROX Bk (24 hr)
Ch 22 - JTable for grid UI (22-3 streams data into map/reduce/JTable model)
Ch 20 - Streaming APIs
FX Grid code example with treeView popup (stackoverflow)
when a table cell is clicked you get a popup with a treeview loaded in it
The Java Language Specification, Java SE 8 (James Gosling et al.)
The Java Tutorials
Site with some useful code examples like logging & File Encryption (uses Java Crypto Extension (JCE) framework)
https://www.reddit.com/r/javaexamples/
Oracle's java8 newtojava book free chapters (many gd examples incl fx)
Introduction to Programming Using Java, 7th ed., August 2014 - dated but lots of gd chapters
(free online) incl threads, networking, generic proramming etc.
Kode Java - website w/code examples like Using zip https://medium.com/codefx-weekly/jsr-376-java-9-optional-multi-release-artifacts-git-fixup-and-jenv-c37d0d69c536
http://jdeferred.org/
https://blog.jevsejev.io/2017/02/19/java-libraries-you-cannot-miss-in-2017/
https://www.javacodegeeks.com/
https://www.mkyong.com/
https://www.infoq.com/
"Java EE is obsolete and even Sprint is bloated. Microservices in docker will be the future."
https://github.com/networknt/light-java-example/tree/master/performance
Unchecked Exceptions; The Controversy The diff betw java.lang.RuntimeException and java.lang.Exception (SO)
Java web frameworks comparison: spring mvc | grails | vaadin | gwt | wicket | play | struts | jsf
http://blog.paralleluniverse.co/2015/05/21/quasar-vs-akka/
https://news.ycombinator.com/from?site=paralleluniverse.co
https://hn.algolia.com/?query=functional%20java&sort=byPopularity&prefix=false&page=0&dateRange=pastYear&type=story
Hashing Implementation Using java.util.ObjectsDZoneJava
by Arun Pandey | Dec. 29, 16 Java Zone
Just released, a free O'Reilly book on Reactive Microsystems: The Evolution of Microservices at Scale.
Brought to you in partnership with Lightbend.
Most applications require an extensive hashing implementation, and writing a good hashing algorithm is
always a tedious job. To get rid of this ugly and tedious hashCode and equals implementation, Java
introduced an API called java.util.Objects in Java 7, which contains a set of utility methods and is useful
in combination with Object instances.
Now let's see an example to understand this cool feature.
Here, Person is a class that needs hashCode() and equals() method implementation.
Prior to Java 7
Using Java 6, we generate the equals() and hashCode() methods using the Eclipse IDE.
Person.java
/**
4 * @author arun.pandey
5 */
6 public class Person {
7 String ssn;
8 String name;
9
10 public Person(){}
11
12 public Person(String ssn, String name){
13 this.ssn = ssn;
14 this.name = name;
15 }
16
17 public String getId() {
18 return ssn;
19 }
20
21 public void setId(String id) {
22 this.ssn = id;
23 }
24
25 public String getName() {
26 return name;
27 }
28
29 public void setName(String name) {
30 this.name = name;
31 }
32
33 /* (non-Javadoc)
34 * @see java.lang.Object#hashCode()
35 */
36 @Override
37 public int hashCode() {
38 final int prime = 31;
39 int result = 1;
40
41 result = prime * result + ((name == null) ? 0 : name.hashCode());
42 result = prime * result + ((ssn == null) ? 0 : ssn.hashCode());
43 return result;
44 }
45
46 /* (non-Javadoc)
47 * @see java.lang.Object#equals(java.lang.Object)
48 */
49 @Override
50 public boolean equals(Object obj) {
51 if (this == obj)
52 return true;
53
54 if (obj == null)
55 return false;
56
57 if (getClass() != obj.getClass())
58 return false;
59
60 Person other = (Person) obj;
61
62 if (name == null) {
63 if (other.name != null)
64 return false;
65 } else if (!name.equals(other.name))
66 return false;
67
68 if (ssn == null) {
69 if (other.ssn != null)
70 return false;
71 } else if (!ssn.equals(other.ssn))
72 return false;
73
74 return true;
75 }
76 }
Java 7 Implementation
Now let's see the same implementation using Java 7 (java.util.Objects) as below.
Person.java
1 package hashingDemo;
2
3 import java.util.Objects;
4
5 /**
6 * @author arun.pandey
7 */
8 public class Person {
9 String ssn;
10 String name;
11
12 public Person(){}
13
14 public Person(String ssn, String name){
15 this.ssn = ssn;
16 this.name = name;
17 }
18
19 public String getId() {
20 return ssn;
21 }
22 public void setId(String id) {
23 this.ssn = id;
24 }
25 public String getName() {
26 return name;
27 }
28 public void setName(String name) {
29 this.name = name;
30 }
31
32 @Override
33 public int hashCode() {
34 return java.util.Objects.hash(getId(), getName());
35 }
36
37 @Override
38 public boolean equals(Object obj){
39 if (obj == this) {
40 return true;
41 }
42
43 if (obj instanceof Person) {
44 Person other = (Person) obj;
45 return Objects.equals(ssn, other.ssn) && Objects.equals(name, other.name);
46 }
47 return false;
48 }
49 }
Test the code to evaluate the above implementation.
HashingTest.java
1 package hashingDemo;
2
3 import org.apache.commons.logging.Log;
4 import org.apache.commons.logging.LogFactory;
5 import org.junit.Test;
6 import junit.framework.Assert;
7
8 /**
9 * @author arun.pandey
10 */
11 public class HashingTest {
12 public static final Log LOG = LogFactory.getLog(HashingTest.class);
13
14 @Test
15 public void testHashing() {
16
17 Person person1 = new Person("xxxx123", "Person1");
18 Person person2 = new Person("xxxx234", "Person2");
19
20 LOG.info("person1 HashCode ==>> " +person1.hashCode());
21 LOG.info("person2 HashCode ==>> " +person2.hashCode());
22
23 Assert.assertFalse(person1.hashCode() == person2.hashCode());
24 Assert.assertFalse("Both are same employee...", person1.equals(person2));
25
26 Person person3 = new Person("xxxx567", "Person3");
27 Person person4 = new Person("xxxx567", "Person4");
28
29 LOG.info("person3 HashCode ==>> " +person3.hashCode());
30 LOG.info("person4 HashCode ==>> " +person4.hashCode());
31
32 Assert.assertEquals(person3.hashCode(), person4.hashCode());
33 Assert.assertTrue("Both are not same employee...",person3.equals(person4));
34 }
35 }
And test passed as expected. Try it in your next hashing and equals implementation to have better and
cleaner code.
I hope this will add a nice feature to your coding. Happy learning!
How and Why to Serialialize Lambdasvanillajava.blogspot.comJava
Overview
Serializing lambdas can be useful in a number of use cases such as persisting
configuration, or as a visitor pattern [https://en.wikipedia.org/wiki/Visitor_pattern] to remote
resources.
For example, so I want to access a resource on a remote Map, I can use get/put, but
say I just want to return a field from the value of a Map, I can pass a lambda as a
visitor to extract the information I want.
MapView userMap =
Chassis.acquireMap("users", String.class, UserInfo.class);
userMap.put("userid", new UserInfo("User's Name"));
// print out changes
userInfo.registerSubscriber(System.out::println);
// obtain just the fullName without downloading the whole object
String name= userMap.applyToKey("userid", u -> u.fullName);
// increment a counter atomically and trigger
// an updated event printed with the subscriber.
userMap.asyncUpdateKey("userid", ui -> {
ui.usageCounter++;
return ui;
});
// increment a counter and return the userid
int count = userMap.syncUpdateKey("userid",
ui -> { ui.usageCounter++; return ui;},
ui -> ui.usageCounter);
As you can see, it is easy to add various simple functions, or call a method to perform
the action you need. The only problem is that lambdas by default are not serializable.
Serializable Lambdas
A simple way to make a lambda serializable is to add a cast of & Serializable to
a variable referring to an implementation of the lambda.
The user of your API doesn't have to explicitly say that the lambda is serializable.
// obtain just the fullName without downloading the whole object
String name= userMap.applyToKey("userid", u -> u.fullName);
The remote implementation serializes the lambda, executes it on the server and returns the result.
Similarly, there is methods for applying a lambda to the map as a whole.
To support queries, you can't use the built in stream() API if you want to implictly add
Serializable. However, you can create one which is as similar as possible.
// print userid which have a usageCounter > 10 each time it is incremented. userMap.entrySet().q
.filter(e -> e.getValue().usageCounter > 10)
.map(e -> e.getKey())
.subscribe(System.out::println);
What makes this different from the regular stream API, is the data could be
distributed across many servers and you get a call back when that changes on any
server. Only the data you are interested in is sent across the network as the filter and
map is being applied on the server.
Java Serialization
Java Serialization is a good generalised, backwardly compatible serialization library.
Two of the most common problems that alternatives try to solve is performance and
cross platform serialization.
In the example above, fullNameFunc serializes to over 700 bytes and there is very
limited options to optimise this to either reduce the size of the message or the amount
of garbage it produces. By comparison, a straight forward binary YAML serialization
uses 348, with more options to optimise the serialization.
This raises the problem of how to serialize a lambda using an alternative, or cross
platform or faster serialization format.
Alternative Serialization
You can hook into the current serialization mechanism. This is not supported, and it
could change at any time but there is not other supported way to do this.
Never the less you can do this
This gives you an object you can inspect to extract the contents of the lambda.
Either to see what method it calls, or to serialize it. On the deserialization side, you
can recreate this object and can readResolve on that object.
Standard API
Currently, there is no standard API for introspection of a lambda. This is done
deliberately so that in future the implementation can be changed, although there is no
public JEP to do so. However, like Unsafe which is internal API, I look forward to the
day when we can use a standard API rather than having to dig into the internals of
the JVM to implement solutions.
Conclusions
With some changes to your API you can make serializing lambdas largely transparent
to the developer. This makes implementing simple distributed systems easier to use
while giving you options to optimise how this is done.
Posted 22nd July 2015 by Peter Lawrey
An interesting HTTP client API is Retrofit by Square. As you've seen, JAX-RS Client can
automatically serialize and deserialize Java objects to JSON objects (or XML).
Retrofit takes this automatic Java/REST translation (which, BTW, is not always a good thing;
domain translations are often particularly leaky abstractions, but they're helpful if you keep
yourself constrained to simple protocols) a step further, and translates the service target
URL - not just the payload - to Java interfaces.
Unfortunately, Retrofit uses the same annotation
names as JAX-RS (server), only defined in a different package, which would make our example
a bit ugly. Luckily, Retrofit has a clone/derivatice called Feign, by Netflix.
The differences between Feign and Retrofit are not entirely clear to me.
Although it seems that Retrofit is more widely adopted (it is older), while
Feign is more easily customizable. In any case, the two are extremely similar,
and can be used pretty much interchangeably
(jackson to decode JSON to/from java)
Feign.Builder feignBuilder = Feign.builder()
.contract(new JAXRSModule.JAXRSContract()) // we want JAX-RS annotations
.encoder(new JacksonEncoder()) // we want Jackson because that's what Dropwizard uses already
.decoder(new JacksonDecoder());
env.jersey().register(new ConsumerResource(feignBuilder));
Our consumer service will now look like this:
@Path("/consumer")
@Produces(MediaType.TEXT_PLAIN)
public static class ConsumerResource {
private final HelloWorldAPI hellowWorld;
public ConsumerResource(Feign.Builder feignBuilder) {
this.hellowWorld = feignBuilder.target(HelloWorldAPI.class, "http://localhost:8080");
}
@Timed
@GET
public String consume() {
Saying saying = hellowWorld.hi("consumer");
return String.format("The service is saying: %s (id: %d)", saying.getContent(), saying.getId());
}
}
mpt: we shd look for current eq using Protocol Buffers/non-JSON/perhaps native binary fmt
Creating / Using RuntimeExceptions
Exceptions
If the client cannot do anything to recover from the exception, make it an unchecked exception.
To create a custom unchecked exception, extend java.lang.RuntimeException
public class ListTooLargeException extends RuntimeException{
public ListTooLargeException(String message) {
super(message);
}
}
For unchecked exception, try and catch the exception is optional.
public class CustomerService {
public void analyze(List data) {
if (data.size() > 50) {
throw new ListTooLargeException("List can't exceed 50 items!"); //runtime exception
} }
the Unsafe
* @throws SecurityException if a security manager exists and its
* {@code checkPropertiesAccess} method doesn't allow
* access to the system properties.
*/
...import jdk.internal.misc.VM;...
@CallerSensitive
public static Unsafe getUnsafe() {
Class caller = Reflection.getCallerClass();
if (!VM.isSystemDomainLoader(caller.getClassLoader()))
throw new SecurityException("Unsafe");
return theUnsafe;
} }
Ancient stuff (2016 & earlier)
Java Platform, Standard Edition (Java SE) 8 - Books
The Java EE Tutorial
The Java Trove:
useful examples of libraries and tools that make working in the Java space a much better experience.
Bld an applic to query a REST-based API and present the results in a JavaFX application.
Part I Part II
Links contained/prods used in Trove->
@Inject
Google Guice alleviates the need for factories and the use of new in your Java code.
Jackson, one of the most well-known JSON parsing libraries out there
jackson-jr, a light-weight (and -featured) alternative to `jackson-databind`: will only deal with "Maps, Lists, Strings, wrappers and Java Beans" (jr-objects), or simple read-only trees (jr-stree)
Java on Azure
...learn how you can build, migrate and scale Java applications on Azure leveraging Azure services, and using tools and frameworks that you know and love...
Java Cloud IDEs:
http://www.koding.com/features
... once you've finalized your app, you'll want to deploy it on a cloud-based service that lets you host apps long term, preferably one with a free hosting option like Heroku.
(looks ok)Free java cloud IDE : https://codetasty.com
also seems good -> https://codeanywhere.com/signup
http://www.koding.com/
ISPs
These days it seems there are servlet engines for nearly every web server, and now we're seeing servlets commonly offered as core value-add for Internet Service Providers (ISPs) and Web Hosting companies.
The megacorp ISPs and Hosting companies still don't offer servlets, though, probably because it's not something Joe Consumer cares about. Also because to host servlets effectively and efficiently requires a bit of expertise. That makes finding a good servlet provider somewhat challenging.
On this page we try to help you on that endeavor by keeping track of all the ISPs that are known to host servlets. For each ISP we provide location and contact information, a short summary, and user reviews. The reviews should help you find out how the ISP you're looking at compares to others in terms of features, reliability, support, and overall score. Feel free to add one of your own.
URLRdr v1_9 updated 9/10/2016 & some srch results generated. URLRdr v1_82 updated 08/16/2016 & some & some srch results generated by it. URLRdr v1_7 (w/timeStamp module) updated 08/08/2016 & lotsa srch results generated by it. URLRdr v1_6 (working GistLink & TinyLink) updated 07/18/2016 URLRdr v1_5 (no soup, no apache includes)
Code for jSoup + escaped JSON string version of URLRdr v1_4.java
//---------------------------------------------------------
FOR TUT POINT SHELL:
http://www.tutorialspoint.com/unix/unix-environment.htm
To set the value of PS1 (PROMPT)
so that it shows the USERNAME & working directory, issue the command -
PS1="[\u@\h \w]\$"
The prompt will show username, machine's name (hostname), and the working directory.
echo $PATH (to see path env) echo $TERM (to see path terminal)
mvn --version (shows maven version, which IS installed on tutpoint)
*SOLVED!
first : find / -name "jsoup*.*"
that listed 2 locations for the jars; incl
/usr/share/java/jsoup/jsoup.jar
/usr/share/maven/lib/jsoup_jsoup.jar
to update env var syntax is: export CLASSPATH= $CLASSPATH:/usr/share/java/jsoup/jsoup.jar
//Before running, will have to compile with jsoup jar:
//export CLASSPATH= $CLASSPATH:/usr/share/java/jsoup/jsoup.jar
//------------------------------------------------------------------------------
import java.net.*;
import java.io.*;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class URLRdr {
public static String getText(String url) throws IOException {
URL website = new URL(url);
String strBody = "";
Document doc = Jsoup.connect (url).get();
Elements bodyTxt = doc.select("body");
for (Element el : bodyTxt){
System.out.println("bodyTxt: "+el);
strBody = strBody + el.toString();
}
return strBody;
}
public static void main( String[] args ) throws IOException{
//---------------------------------------------------------
//URLRdr.java (for tutorialspoint.com)
import java.net.*;
import java.io.*;
public class URLRdr {
public static String getText(String url) throws Exception {
URL website = new URL(url);
URLConnection connection = website.openConnection();
BufferedReader in = new BufferedReader( new InputStreamReader( connection.getInputStream()));
StringBuilder response = new StringBuilder(); String inputLine; while ((inputLine = in.readLine()) != null) response.append(inputLine);
in.close();
return response.toString();
}
public static void main(String[] args) throws Exception {
String content = URLRdr.getText(args [0]);
System.out.println(content);
}
}
//---------------------------------------------------------
//java URLRdr "https://www.yahoo.com/? p=dnr" >
HelloWorld.java
//then refresh the project to see HelloWorld.java which's auto added to Project
Search param. notes: To get 30 result on single page, pass below parameter:
For bing use count=30 - http://www.bing.com/search?q=test&count=30
For yahoo use n=30 - https://search.yahoo.com/search? p=test&n=30
For google use num=30
- https://www.google.com/search?q=test&num=30
Interesting libraries (many've been updated since last checked)
jimfs: An in-memory file system for Java 7+, poss updated since last use
Gulava allows you to write relational predicates in Java. You can write Prolog-style predicates and use them from normal Java code, seamlessly integrated with the magic of annotation processors.
(uses annotations to generate preds; class Goal accepts Objects & pred is blt frm that)
Reagera is a minimal implementation of a predictable state container in Java, inspired by Elm and Redux.
http://couchdb.apache.org/
Apache CouchDB is open source database software that focuses on ease of use
and having an architecture that "completely embraces the Web".[2] It has a
document-oriented NoSQL database architecture and is implemented in the
concurrency-oriented language Erlang; it uses JSON to store data, JavaScript
as its query language using MapReduce, and HTTP for an API.[2]
http://couchdb.apache.org/fauxton-visual-guide/index.html
PDF
HN Thread: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
I don't understand why developers use cloud for bootstrapping/side project. Digital Ocean is all you need $5 droplet + $15 Postgres or even better $7 dyno on Heroku.
...
Heroku costs are pretty predicable, and you can easily set a maximum scalability threshold to their auto-scalable dynos, so that they will never cost you more than a predefined amount of money
>>I'm just a student but I've spent about 10 hours trying to figure out why Azure >>has been charging me >$5/day for their "basic" database @5DTUs, 2gb max >>storage.
Do you have geo-replication turned on? More regions will be an additional $5/month (plus bandwidth between regions) if you replicate. You can serve everything out of a single region but it is pretty easy to add others if you're not paying attention during initial setup.
Yeah a friend of mine wanted a real cert, not letsencrypt (I don't understand how that is more real but ok), as a bit of a noob he clicked around on the AWS website and some days later had a bill op 1500 eur. They also nulled it. Still, this scares the hell out of me.
...we're in the process of switching from Digital Ocean to Hetzner for a project, that will increase infrastructure performance (roughly memory/cpu/storage) by 4x and decrease costs by 4x. And no driving to the colo center is neccessary, as it's their dedicated server, so their on-site engineers do the hardware replacement.
,,,writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk.
I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs would be absolutely fixed and known.
On a tight budget? You don't even need to go as far as $1200, I see totally fine test/development environment suitable, Dell 1U servers on eBay right now for under $500 with 128GB of RAM.
Or that would be better off purchasing a fixed-configuration virtual machine (typically running on xen or kvm underneath) that has a certain specific amount of CPU, RAM and storage resources allocated to it which cannot balloon. For a fixed bill per month like $65 or $85.
>>Can anyone recommend a similar provider but for North America?
https://oneprovider.com/dedicated-servers-in-north-america
...Always use a virtual card for this. This way you can tell in advance how much can be drawn max and the expiration date.
...Instead I use DigitalOcean where you have droplet limits that you can set, and the ability the pre-pay if you pay by PayPal, and never enter my bank card
...I use DigitalOcean and not GCP/AWS, personally and professionally, for the exact same reason: they provide a way to not give the company a "free for all" access to a bank acccount.
...I tried to signup with a gift card and it was rejected.
I can attach a gift card to paypal and use it.
https://github.com/fakemongo/fongo Fongo is an in-memory java implementation of MongoDB. It intercepts calls to the standard mongo-java-driver for finds, updates, inserts, removes and other methods. The primary use is for lightweight unit testing where you don't want to spin up a mongod process.
Capacity planning:...Additional capacity can be added with no application downtime, and you can optionally enable auto-scaling to respond automatically to spikes in usage.
Ensuring that applications gracefully handle cluster failover through testing.
"ALL indexes shd fit in memory" If your index doesnt fit in memory the slow down you will experience when querying on that index as pages are read from disk expired and then reread from disk will be dramatic.
The goal for every performant database is to have all its regularily queried indexes to fit completely in RAM. To futher increase performance the most regularily queried records should fit in RAM as well.
For a gaming application this means data associated with logged in users, for a banking application accounts that have transactions in the last thirty days etc. etc.
...from mongoDevBlog related to above, how to fit all idx es into RAM... In MongoDB Collections and Indexes both use the same underlying data structure. WiredTiger tables - these are BTrees (B+ ish trees) , essentially key-value stores arranged into pages(or blocks depending who you ask) - each page is 32KB in size or one entry whichever is larger (Up to 16MB for a single 16MB BSON document) - its then compressed when flushed to disk. In memory a collection page contains both the the on-disk version of the page as well as a list of changes to any document not yet reconciled to disk. An Index page contains the index keys and for each key (which may be the full key or prefix compressed) a pointer to the identity of the document it points to.
When you access an index - MongoDB checks to see if the page is in cache - if not it reads it from the disk (going via the OS pagecache so it may be cached there) - index pages are not compressed on disk as they already have prefix compression.
The upshot is that each index lookup may in cache (nice and fast) or not in which case , nothing to do with page faults - this is all explicitly tracked, it will require a disk read which will be 32KB and a seek at the very least - if you have readahead set appropriately it may be more than 32KB. If that happens to be in your OS page cache it will be quicker but it still needs some processing to put it in cache. The seek will take 1 IO operation at least so a 1000 IOPS disk, with random seeks in an index which is much larger than ram will be very much throttled by IO.
You can look at the Read-Into-Cache metric using any of the MongoDB tooling (or look in db.serverStatus() and diff the entries) you can also observer the cache for collections and indexes using mongocacheview.js (https://github.com/johnlpage/MongoCacheView).
In general - if your working set does not fit in ram, expect one to two orders of magnitude slower operations that hit those keys (queries, inserts, deletes, updates of indexed values) , consider (a) Adding RAM (b) Making indexes smaller Adding many IOPS - in that order.
...also... If you want to preheat (upload it into RAM) your indexes when mongod start you may use db.runCommand({"touch" : , "data" : true, "index" : true}). Accessing index entries that are not in memory is particularly inefficient, as it often causes two page faults. There is one fault to load the index entry into memory and then another to load the document into memory. When an index lookup causes a page fault its called a btree miss. MongoDB also tracks btree hits: when an index access does not have to go to disk (from MongoDB definitive guide). Alexey Smirnov Dec 31 '16 at 12:36
>>$or query using more than 2 indexes takes forever Joris_SEBIRE Hello, I have a probleme with a mongo query that uses an $or like : { a: 1, b: 2, $or: [ { c: 3 }, { d: 4 }, { e: 5 }, { f: 6 } ] } i have every a_b_c, a_b_d, a_b_e and a_b_f indexes but that query takes more than 150 seconds. But, if i comment any 2 of the 4 items in my $or, the request takes 0.5 seconds Does mongo applie a limit of index used in an $or query ? or something i didnt get ? Thanks for your answer.
Pavel_Duchovny MongoDB Employee Welcome to MongoDB community. The ways your indexes are built the engine tries to do an index intersection loading all indexes to memory and merging them. I think that for this query its best to use either all fields in one index or just index a,b Thanks, Pavel
Sudhesh_Gnanasekaran Refer to this documentation that might be helpful for your use-case. https://docs.mongodb.com/manual/core/index-compound/ ,...from above...MongoDB imposes a limit of 32 fields for any compound index. Compound indexes can support queries that match on multiple fields. ...Index fields are parsed in order; if a query omits a particular index prefix, it is unable to make use of any index fields that follow that prefix.
...The choice between creating compound indexes that support your queries or relying on index intersection depends on the specifics of your system. (see https://docs.mongodb.com/manual/core/index-intersection/#index-intersection-compound-indexes)
MongoDB may use the same wildcard index for satisfying each independent argument of the query $or or aggregation $or operators.
if you want the aggregation to use the index you need to perform either a $match or $sort stages in the beginning of the query. eg, db.collection.aggregate([{$match:{"Instance.field":"value"}, {$unwind: "$Instance"}, {$match:{"Instance.field":"value"}}])
https://stackoverflow.com/questions/8607637/are-there-any-tools-to-estimate-index-size-in-mongodb mongodb query and index optimization.
groupBy note: "$group does not order its output documents." See http://docs.mongodb.org/manual/reference/operator/aggregation/group/
$limit limits the number of processed elements of an immediately preceding $sort operation, not only the number of elements passed to the next stage. See the note at http://docs.mongodb.org/manual/reference/operator/aggregation/limit/
>>pagination on group data mongodb -
in $group items you can't directly apply pagination, but below trick will be used , step 1 - write aggregation on product table, and write groupBY step 2 - prdSkip for skipping , and limit for limiting data , pass it dynamically finally query looks like - params - limit , skip - for category pagination and prdSkip and PrdLimit for products pagination db.products.aggregate([ { $group: { _id: '$prdCategoryId', products: { $push: '$$ROOT' } } }, { $lookup: { from: 'categories', localField: '_id', foreignField: '_id', as: 'categoryProducts', }, }, { $replaceRoot: { newRoot: { $mergeObjects: [{ $arrayElemAt: ['$categoryProducts', 0] }, '$$ROOT'], }, }, }, { $project: { // pagination for products products: { $slice: ['$products', prdSkip, prdLimit], }, _id: 1, catName: 1, catDescription: 1, }, }, ]) .limit(limit) // pagination for category .skip(skip); answered Oct 22 '20 at 6:50 Akshay Dhawle
<>
$bucket Categorizes incoming documents into groups, called buckets, based on a specified expression and bucket boundaries.
>>The example for $bucket uses push to create a "bucket" w/an aggregation
containing an array of Docs; GroupedBy specific crit. @ https://docs.mongodb.com/manual/reference/operator/aggregation/bucket/#pipe._S_bucket
Use the $accumulator operator to execute your own JavaScript functions to implement behavior not supported by the MongoDB Query Language See also $function.
MongoCollection myCollection = myDB.getCollection("myCollection");
Map multiIdMap = new HashMap();
multiIdMap.put("groupField1", "$groupField1");
multiIdMap.put("groupField2", "$groupField2");
Document groupFields = new Document(multiIdMap);
AggregateIterable aggregate = myCollection.aggregate(Arrays.asList(
Aggregates.group(groupFields,
Accumulators.last("lastDate", "$dateCreated"),
Accumulators.last("lastNumAvail", "$availableUnits")
)
));
Mongo aggregation in java: group with multiple fields
MongoDatabase mongo = // initialize your connection;
Document matches = new Document("$match",
new Document("gi", new Document("$ne", null))
.append("ci", Integer.parseInt(courseid)));
Document firstGroup = new Document("$group",
new Document("_id",
new Document("ci", "$ci")
.append("gi", "$gi")
.append("gn", "$gn")
.append("si", "$si"))
.append("count", new Document("$sum", 1)));
Document secondGroup = new Document("$group",
new Document("_id",
new Document("ci", "$_id.ci")
.append("gi", "$_id.gi")
.append("gn", "$_id.gn"))
.append("ns", new Document("$sum", 1)));
Document sort = new Document("$sort",
new Document("_id.gi", 1));
for(Document doc : cursor) { // do stuff with doc }
Aggregation.group(Fields.fields()
.and("field1")
.and("field2"))
.first("name")
.`as`("name")
.count().`as`("count")
This will produce the following MDB query:
2.12: Pagination in mongoDb Collections MPT note: IF NECC this approach can be used by
- adding a new tmp Counter field *AFTER GROUPING*
- finding the range#s as above; then getting the pg.
SO post: When talking about pagination in MongoDB, it is easily to write this code:
collection.find().skip(pageSize*(pageNum-1)).limit(pageSize);
Above is the native solution supported by MongoDB, but this is not efficient if there are huge documents in the collection. Suppose you have 100M documents, and you want to get the data from the middle offset(50Mth). MongoDB has to build up the full dataset and walk from the beginning to the specified offset, this will be low performance. As your offset increases, the performance keeps degrade.
The root cause is the skip() command which is not efficient and can not take big benifit from index.
Below is another solution to improve performance on large data pagination:
The typical usage scenario of pagination is that there is a table or list to show data of specified page, and also a 'Previous Page' & 'Next Page' button to load data of previous or next page.
If you got the '_id' of the last document in current page, you can use find() instead of skip(). Use _id > currentPage_LastDocument._id as one of the criteria to find next page data. Here is pseudocode:
//Page 1
collection.find().limit(pageSize);
//Get the _id of the last document in this page
last_id = ...
//Page 2
users = collection.find({'_id': {$gt: last_id}}).limit(pageSize);
//Update the last id with the _id of the last document in this page
last_id = ...
This will avoid MongoDB to walk through large data when using skip().
edited Jun 4 '19 at 12:17
SQB answered May 10 '18 at 6:47
Card Games
Google public domain playing cards image set (PNG/SVG)
Chess
pgn spec
Tons of pgn games by player/opening
Online FEN position viewer
Garden of Chess Openings
Annotated Games repo
WikiMedia chess icons C & C++ versions of pgn2fen
F# chess progs FsChessPgn: a library for chess scripting using F#
FsChessPgn is a library for scripting in F# that provides chess facilities such as move generation, move validation and support for common chess formats such as FEN and PGN.
Docs
Generates images of a Board in PNG format.
Can make moves given a Board. Can both make and unmake moves given a Game.
Show a simple ASCII board | Detects checkmates and stalemates | Detects checks and attacks
Reads and writes PGNs. Supports headers, comments, NAGs and a tree of variations
FsharpChess Chess with an F# domain engine and a simple WPF UI.
Console version (C#) using emoji glyphs
FsChess A toy chess engine written in F#. So far it only implements basic moves, more complicated moved like en passant and castling are still to do
Assemblies Version checking only occurs with strong-named assemblies.
You can also Limit/require assembly bindings to a specific version, etc.
To use two or more versions of an assembly in the same application (extern alias); e.g.
(cmdLine) /r:GridV1=grid.dll /r:GridV2=grid20.dll
This creates the external aliases GridV1 and GridV2; use via GridV1::Grid or GridV2::Grid
Set assembly attribs via (inline in code) ->
[assembly:AssemblyVersionAttribute("2.0.1")] //major.minor.build.revision
AssemblyCompanyAttribute("company name")
AssemblyCopyrightAttribute("copyright information")
AssemblyProductAttribute("product information")
AssemblyDescriptionAttribute("") nature and purpose of the assembly.
AssemblyDefaultAliasAttribute("") <-String specifying a default alias to be used by referencing assemblies. can also be used as a short form of the full assembly name.
AssemblyTitleAttribute("") <-String specifying a friendly name for the assembly. For example, an assembly named comdlg might have the title Microsoft Common Dialog Control.
(old stuff from the other bookmarks file) f# .netCore
.NET documentation | Microsoft Docs
.NET introduction and overview | Microsoft Docs
Building a WPF application in functional way | F# Snippets
Develop libraries with the .NET CLI - .NET | Microsoft Docs
F# - Do It All with F# on .NET Core | Microsoft Docs
F# - SAFE Stack: Functional Web Programming for .NET Core | Microsoft Docs
F# coding conventions | Microsoft Docs
F# docs - get started, tutorials, reference. | Microsoft Docs
java2c# Commercial, free demo
paulirwin/JavaToCSharp Sharpen (java to c#)
Unsupported APIs on .NET Core and .NET 5+ | Microsoft Docs
Using F# on Azure | Microsoft Docs
What's new in .NET 5 | Microsoft Docs
JLCA is available now - Visual Studio | Microsoft Docs
fs snip nice Memoize example
https://github.com/mikhailshilkov/azure-functions-fsharp-examples
Regexp (straight net api calls):
let TestConnString =
""""""
Regex.Match(connectionString, "[dD]atabase=(?.*?);").Groups.["database"].Value
SecureString
For console.readPwd; look @ Security.SecureString;
pump into Marshal.SecureStringToGlobalAllocUnicode
then to Marshal.PtrToStringUni & finallly Marshal.ZeroFreeGlobalAllocUnicode
match (Console.ReadKey true).KeyChar
with
| '\r' | '\n' ->
Console.WriteLine ()
charList
| chr ->
Console.Write '*'
readKey <| chr :: charList
CSV parsing
// An example of how you might manually parse a CSV file in F#.
open System
open System.IO
Environment.CurrentDirectory <- __SOURCE_DIRECTORY__
let file = @"..\..\data\FootballResults.csv"
type Result =
{ Date : DateTime
Home : string
Away : string
HomeGoals : int
AwayGoals : int }
let data =
file
|> File.ReadAllLines
|> Seq.skip 1
|> Seq.map(fun row ->
let fields = row.Split ','
{ Date = DateTime.ParseExact(fields.[0], "MM/dd/yyyy", null)
Home = fields.[1]
Away = fields.[2]
HomeGoals = int fields.[3]
AwayGoals = int fields.[4] })
data
|> Seq.filter(fun row -> row.HomeGoals > row.AwayGoals)
|> Seq.countBy(fun row -> row.Home)
|> Seq.sortByDescending snd
|> Seq.take 3
List fold snippet from EventSourcing-DIY
module List
/// Map a Result producing function over a list to get a new Result
/// ('a -> Result<'b>) -> 'a list -> Result<'b list>
let traverseResult f list =
// define the monadic functions
let (>>=) x f = Result.bind f x
let retn = Result.Ok
// right fold over the list
let initState = retn []
let folder head tail =
f head >>= (fun h ->
tail >>= (fun t ->
retn (h :: t) ))
List.foldBack folder list initState
fslang-suggestions: Allow Unicodesymbols to be used as operators -> has a working workaround
Unicode Symbol Table
Fpish: Event.guard to run Async if race conditions
Event.guard takes a lambda and an event, and returns that same event except it will run the lambda after subscribing so that you never 'miss' the event if the lambda may provoke it.
Composing validations together to parse a csv file (no invalid inpt allowed via only valid types resulting.)
(DSyme) Operators You Might Need to Know f1 >> f2 Function composition (expr1, expr2) ||> f Two-argument pipelining (expr1, expr2, expr3) |||> f Three-argument pipelining
n..step..m Range with step (within a list or sequence or loop)
let (|A|_|) arg = … Defining active patterns let (|A|B|) arg = … Defining active patterns
Operators You Don't Need to Know !cell Dereference a mutable reference cell. Use cell.Value instead := Assign a mutable reference cell. Use "cell.Value <- expr" instead @ Append one list to another. Can normally use a computed list expression or List.append instead
<< Backward function composition, discouraged in favour of forward composition
<| Back-piping, discouraged in favour of forward piping <|| Back-piping, discouraged in favour of forward piping <||| Back-piping, discouraged in favour of forward piping
*?, +? … Nullable operators (these are used exceptionally rarely in LINQ queries, ignore these)
Should I Define My Own Operators? Almost certainly not. See msft design guidelines
Reflection on Nested Types in DU
open System
type OptModID = | OptModID of i_modNm: string
type DocVersionModule = | DocVersionModule of OptModID * lastVerNum:int * hasPriorVers:bool with
member this.desc() =
let (DocVersionModule(OptModID(modNm), lastVerNum, hasPriorVers)) = this
"Optional Module: " + modNm +
match hasPriorVers with
| true -> " has PriorVersions = true; lastVersionNumber: " + lastVerNum.ToString()
| _ -> " doesn't have PriorVersions"
let oldM = DocVersionModule( OptModID("ver 2") , 2, false)
type CustomVer =
| DocVersionModule of DocVersionModule
| CustomVer of DocVersionModule * newFld:string
let oldM2 = CustomVer.DocVersionModule(oldM)
let newM = CustomVer(oldM, "newFld")
let li:list<_> = [oldM2; newM]
List.map(fun (x) ->
match x with
| DocVersionModule m -> printfn "found old mod:%A " m
| _ -> printfn "found new mod:%A " x
) li |> ignore
> typeof.GetNestedTypes()
|> Seq.iter (fun t ->
let p = t.GetProperties()
let s =
p
|> Array.map (fun p -> sprintf "%s: %s" p.Name p.PropertyType.Name)
|> String.concat "; "
printfn "Nested type %s: %i Properties %s" t.Name p.Length s
)
Also, from quotations.fs (f# repo)
let getUnionCaseInfoField(unionCase:UnionCaseInfo, index) =
let fields = unionCase.GetFields()
if index < 0 || index >= fields.Length then invalidArg "index" (SR.GetString(SR.QinvalidCaseIndex))
fields.[index]
Unlicense or cc0-1.0(no attrib) f# projects
Github srch query Chessie.ErrorHandling (Railway-oriented programming)
FSharpx.Extras
has Reader/Writer/Async, Regexp stuff & some other stuff, gd examples
FSharp.Quotations.Evaluator.Tools (contains 7tpl & expression stuff)
fsprojects/FSharp.Compiler.CodeDom (has a CodeDom Visitor and compiler for ides
REF: This contains a file 'waitUntilExists' func:
https://github.com/microsoft/fsharplu/blob/master/FSharpLu/File.fs
(use w = new System.IO.FileSystemWatcher(parentDir, fileName, IncludeSubdirectories = false, EnableRaisingEvents = true))
REF: Azure Db locally stored (using LiteDb) entirely in f#, with emulator
fsharplu: MIT License (displ lic on substantial reuse)
/// State machine-based asynchronous agents
/// ... supports fork/join, continuations
namespace Microsoft.FSharpLu.Actor.StateMachine // This module implements a lightweight Http server using HttpListener.
/// The purpose of the server is to enable inter-process communication
module Microsoft.FSharpLu.HttpCommunication
Brian McNamara (msft)'s Blog:
Graphing a subReddit using F# & DGML
Matt Thornton's Monad Series
https://geekeh.com/
https://thinkbeforecoding.com/
https://www.wtfsharp.net/hosts/stachu
https://devonburriss.me/tag/fsharp/
SO threads on F# (some selected; + more remain to chk)
https://medium.com/hackernoon/reflecting-on-f-in-2017-5ac67fb138ff ->
* Suave and Giraffe emerged as two dominant libraries to use when writing web services on .NET Core, with Freya as a brilliant alternative that everyone should try out as well.
* Fable rapidly evolved from an interesting project with potential into an impressive and fully-fledged alternative for JavaScript in the browser, allowing you to write Full-Stack F# applications.
New('21 May04) F# reading from HN
I just sat in a Mathematical Planning seminar with a guy from Quicken Loans who uses F# in production at work (Matthew Crews). He is also an author behind the Flips F# library.
A quick list of F# libs for real world, everyday uses:
* Pulumi / Farmer - code as infrastructure
* FAKE - build tool
* Fable - javascript transpiler
* Feliz - react elements
* FSharp.Data - parsing csvs, html pages
http.fs : A simple, functional HTTP client library for F#
Note:
Poss. reason behind the @MBIs trying to force 'earlier version' adoption ->
,,,(if you're targetting) Mono assemblies ... do lag .NET CLR in functionality, so
you'll probably need to use an F# version a few minor releases earlier
than the official one.
(So if they want to steal/use code elsewhere (bastas prefer macs) they'd have to... )
F# using companies: Jet did, but they're now dead...or do they still live on inside of the greater Walmart (Labs?) organization?
Their F# projects under github.com/jet are still active
Quant finance: G-Research
Q: Can anyone recommend a a project or two that is small enough for a newcomer to wrap their minds around, that also exemplifies what makes a high quality F# codebase?
2021 2017 2016 * Compositional IT has a lot of good blog posts on how they use F# to solve real world problems.
* Kit Eason's Stylish F# is a good book.
HN F# threads (only read one)
Notes from Podcasts
Rachel Reese (Jet)
300+ microsvcs in F#
TypeProv.s with CSV files
Heavy Xamarin.Forms usage
Bob Martin (Uncle Bob)
TopDnDev 4 mocks (sketch out top 1st instd of botm)
A combinator is a higher-order function that uses only function application and earlier defined combinators to define a result from its arguments.
inv. (1920)...Schnfinkel's system was essentially equivalent to a combinatory logic based upon the combinators
B
-- | B combinator - bluebird - Haskell ('.').
bluebird :: (b -> c) -> (a -> b) -> a -> c
bluebird = (.)
,
C
-- | C combinator - cardinal - Haskell 'flip'.
cardinal :: (a -> b -> c) -> b -> a -> c
cardinal = flip
,
I,
K
-- | K combinator - kestrel - Haskell 'const'.
-- Corresponds to the encoding of @true@ in the lambda calculus.
kestrel :: a -> b -> a
kestrel = const
, and
S
-- | S combinator - starling.
-- Haskell: Applicative\'s @(\<*\>)@ on functions.
-- Substitution.
starling :: (a -> b -> c) -> (a -> b) -> a -> c
starling f g x = f x (g x)
,
. Schonfinkel was able to show that the system could be reduced to just K and S (i.e., SKI calculus) and outlined a proof that a version of this system had the same power as predicate logic.
His paper also showed that functions of two or more arguments could be replaced by functions taking a single argument (i.e., Currying)
Haskell Aviary (more compl.) Bird list raganwald
... more @ his backup repo; addresses Ruby/js but v. useable ...
(raganwald) Given fn; {do! (side-effect) (fn) (side-effect) }
A combinator that takes a function and imposes some
side effects before and after the function
e.g.,
Free memory (C) after using it (B) whenever we have allocated it (A)
or
open files/sockets (A), reading/writing (B), then closing them (C)
or
accessing mutexen (A), working therein (B), and leaving (C)
A composition of Kestrel and Thrush gets the job done:
K(Txy) = Kyx = y
K(Txyz) = Kyxz = yz
K(K(Txyz)) = K(Kyxz) = Kyz = y
(raganwald) Kestrels for DSL HardDrive.new.capacity(150).external.speed(7200)
Instead of:
hd = HardDrive.new
hd.capacity = 150
hd.external = true
hd.speed = 7200
OR instead of
def fizz(arr)
arr.pop
arr.map! { |n| n * 2 }
end
We can write:
def fizz(arr)
arr.tap(&:pop).map! { |n| n * 2 }
end
also, (@ lst in Ruby) K:Object#dont is the Ruby-semantic equivalent of commenting out a method call, only it can be inserted inside of an existing expression, eg.
JoinBetweenTwoModels.dont.create!(...) do |new_join|
(raganwald) Using a 3-param-C as a fn generator The Cardinal is written Cxyz = xzy. In Ruby:
cardinal.call(proc_over_proc).call(a_value).call(a_proc)
=> proc_over_proc.call(a_proc).call(a_value)
This implies that proc_over_proc is a function that takes a function as its argument and returns a function; i.e., you want a cardinal when you would like to modify what a function does.
(raganwald) Using the
Quirky -- | Q3 combinator - quirky bird. quirky :: (a -> b) -> a -> (b -> c) -> c quirky f x g = g (f x) Bird The quirky bird is written Q3xyz = z(xy). In Ruby:
quirky.call(value_proc).call(a_value).call(a_proc)
=> a_proc.call(value_proc.call(a_value))
Like the cardinal, the quirky bird reverses the order of application. But where the cardinal modifies the function that is applied to a value, the quirky bird modifies the value itself.
ie, Q addOne 2 squareFirst = 5
Cont Monad
School of Haskell:The Mother of all Monads (Dan Piponi)'s exc. intro; refers to his post
per riptutorial.com; async in F# is a Continuation Monad; A coroutine Monad.
from brian McNamara(msft)'s blog (catamorphisms series):
type ContinuationBuilder() =
member this.Return(x) = (fun k -> k x)
member this.ReturnFrom(x) = x
member this.Bind(m,f) = (fun k -> m (fun a -> f a k))
member this.Delay(f) = f()
let K = new ContinuationBuilder()
This def isn't congruent to Haskell's (since they use the predefined m)
...(I) wrote CallCC at the representation level; I think this is maybe also the "traditional" approach for such monads...
(see discussion on fpish)
Some other definitions:
Scott Wlaschin's article on continuations with CE.
https://github.com/ocaml-batteries-team/batteries-included/
here's the batInterface.ml def of Monad:
module type Monad = sig
type 'a m
val bind : 'a m -> ('a -> 'b m) -> 'b m
val return: 'a -> 'a m
end
Here's how oCaml defines State (reversed) ->
(Chattered/ocaml-monad) ln # 473
module State(T : sig type s end) =
struct
include Make(struct
type 'a m = T.s -> (T.s * 'a)
let return x s = (s, x)
let bind xf f s =
let s',x = xf s in
f x s'
end)
let read s = (s,s)
let write x s = (x,())
let run x s = x s
let eval x s = snd (x s)
let modify f = bind read (fun s -> write (f s))
end
Threading/ IO monad vs threads: the case of web-app servers
'Interfacing mutable operations within a functional core - program design'
Wikipedia on Generalizations of monads:
Arrows use additional structure to bring plain functions and monads under a single interface
Monad transformers act on distinct monads to combine them modularly
Alternatives for modeling computations:
Effect systems are a different way to describe side effects as types
Uniqueness types are a third approach to handling side-effects in functional languages (mpt: no gd.)
from the F# Repo:
This has a fn traversing/using MANY quotPatterns -> Quotations debug Here are all the patterns (Line#2251) quotations.fs
The three main things you should know about the quotations API
(from Traversing and transforming F# quotations: A guided tour)
Quotations.Expr and Quotations.Patterns are each others dual: they are used for constructing and deconstructing Expr types respectively;
Quotations.ExprShape contains the workhorse of any quotations-using implementation: a zipper-like pattern for traversing and transforming any quotation in a straightforward way
Splicing is used to effectively and readably construct parameterized expressions as an alternative to Expr.Whatever calls.
// tryFindOne to get an Option<'t>
let john = people.tryFindOne q fun person , person.Name = "John" @>
// findOne to get a single existing document
let firstAlbum = albums.findOne <@ fun album -> album.Id = 1 @>
// findMany to get many documents of seq<'t> that match the query
let popularAlbums = albums.findMany <@ fun album -> album.IsPopular @>
// using "not" function let unverifiedEmails = accounts.findMany <@ fun ace -> not acc.EmailVerified @>
// with comparison operators
let albumsWithManySongs = albums.findMany q fun album -> album.SongCount > 50
// composite operators
let popularAndHaveManySongs = albums.findMany q fun album -> album.IsPopular && album.SongCount 50
// full search with a customized predicate
let releasedThisYear = albums.fullSearch <@ fun album -> album.DateReleased @> (fun dateReleased -> dateReleased.Year = DateTime.Now.Year)
Dec'21 Task: @Rsch ex quot work for feeding string to cmp/eval This suffices -> (uniLicense; based on the old PowerPack)
https://fsprojects.github.io/FSharp.Quotations.Evaluator/tutorial.html
On-the-fly code generation
You can generate lambdas and compile them dynamically:
let tupler =
let v = Var("x",typeof)
let v = Expr.Lambda(v, Expr.NewTuple [Expr.Var v; Expr.Var v]) // codegen (fun x -> (x,x))
v.CompileUntyped() :?> (int -> (int * int))
tupler 78 // (78, 78)
I can't talk about Clojure since I have't used it, but I know a bit about DSLs in F#. F# provides two main language oriented programming features (which is what Don Syme likes to call them): code quotations and computation expressions.
Code quotations are closer to what you would get with macros in a language like Lisp. They allow you to generate expressions programmatically which you can then execute. By using the ReflectedDefinition attribute on F# expressions you get access to their ASTs. See msdn for more details.
(this is one of the guys bhd FSharpx)
(using Visitors + a dsl-like (from scratch) evaluator for simple logicOps
cn be extended...) See also Juliet Rosenthal's SO post mentioned.
let f = Option.map sqrt << Option.filter ((<=) 0.)
...Eventually you end up in point-free world, where foo|>bar|>baz and so on.
.net5 PBE api CodeProj: emulating PBE under .net
.net/Java interop AES w IV
crypto in C & Java
InterOp encr in Java/dotNet
Translate PBE from Java to C A collection of common (interesting) cryptographic mistakes and learning resources.
Representing SHA-256 Hashes As Avatars related: generate unique images from any text (above art also refers to github/goog using this methd)
no line#s...takes whatever you enter and wraps it in a Main method which is then wrapped in a class
F# Compiler service examples (has both FSharpSyntaxTokenKind.Function & FSharpSyntaxTokenKind.Val)
Tokenizing / Lexing src. code
https://fsharp.github.io/FSharp.Compiler.Service/index.html
Using Roslyn to parse the AST.
fcs samples Tokenization example (uses & requires FSharp.Compiler.SourceCodeServices)
fsprojects snippet which uses Fsharp.Formatting.CodeFormat.dll to tokenize F# source.
Understand the .NET Compiler Platform SDK model
MSFT Cosmo Froto (protoBuffs) Db app (Suave + Mongo) Falanx by jet.com (now wm)
Code generation from Protobuf v3 schema (rather than types being injected as a type provider.)
Mondocks: An alternative way to interact with MongoDB databases from F# that allows you to use mongo-idiomatic constructs
Cobalt Blue is: hex -> #0047AB; decimal RGB -> rgb(0,71,171)
Royal Blue is: hex -> #00275E (see: http://www.99colors.net/hex/00275E)
rgba(0, 39, 94, 1); hsla(215, 100%, 18%, 1)
Same color in lighter hues: #3365AA, #5E8ED1
Related blues (lightest is #3):
1. Air Force blue (Hex code #5E8ED1)
2. Lime (Hex code #BFDAFF)
3. Platinum (Hex code #E5F0FF)
(Ancient) Forms example with somewhat unusual UI idioms (for us)
Handler->
Line# 320: let rec ComboSelectHandler (sender:obj) (e:EventArgs) =
... in
let _ = combo.SelectionChangeCommitted.AddHandler( new EventHandler(ComboSelectHandler) ) in
folderPanel.Controls.Add(combo)
Props ->
let labelStart = new Label()
do labelStart.Anchor <- AnchorStyles.Bottom ||| AnchorStyles.Right
do labelStart.Text <- "Start"
do rangeTable.Controls.Add(labelStart)
Menu->
let memuMain = mainForm.Menu <- new MainMenu()
let menuFile= mainForm.Menu.MenuItems.Add("&File")
let miFileOpen = new MenuItem("&Open")
let miFileExit = new MenuItem("&Exit")
do menuFile.MenuItems.Add(miFileOpen) |> ignore
do menuFile.MenuItems.Add(miFileExit) |> ignore
do miFileOpen.Click.Add( fun _ -> ...
WinForms snippet->
open System
open System.Windows.Forms
[<EntryPoint>]
[<STAThread>]
let main argv =
Application.EnableVisualStyles()
// Application.SetCompatibleTextRenderingDefault(false)
let form = new Form(Width= 400, Height = 300, Visible = true, Text = "Hello World from F#!")
form.TopMost <- true
form.Click.Add (fun args-> printfn "the form was clicked")
Application.Run(form)
0
WinForms AnimationSO: How can I add moving effects to my controls in C#?
...I have a Panel In my C# form and I have a button. When I click on the Button the invisible Panel Shows. Instead of that I want the Panel to move in or slide in.
Approach #1
A>>Window animation is a built-in feature for Windows. Here's a class that uses it:
using System;
using System.ComponentModel;
using System.Windows.Forms;
using System.Runtime.InteropServices;
public static class Util {
public enum Effect { Roll, Slide, Center, Blend }
public static void Animate(Control ctl, Effect effect, int msec, int angle) {
int flags = effmap[(int)effect];
if (ctl.Visible) { flags |= 0x10000; angle += 180; }
else {
if (ctl.TopLevelControl == ctl) flags |= 0x20000;
else if (effect == Effect.Blend) throw new ArgumentException();
}
flags |= dirmap[(angle % 360) / 45];
bool ok = AnimateWindow(ctl.Handle, msec, flags);
if (!ok) throw new Exception("Animation failed");
ctl.Visible = !ctl.Visible;
}
private static int[] dirmap = { 1, 5, 4, 6, 2, 10, 8, 9 };
private static int[] effmap = { 0, 0x40000, 0x10, 0x80000 };
[DllImport("user32.dll")]
private static extern bool AnimateWindow(IntPtr handle, int msec, int flags);
}
Sample usage:
private void button2_Click(object sender, EventArgs e) {
Util.Animate(button1, Util.Effect.Slide, 150, 180);
}
Hans Passant
Q:Does this only work on contained controls, Hans?
A:I tested it on contained controls. It should work on toplevel forms as well, however it is going hide the form on the first call since you'd have called Show() to display it. Replacing Show() so it animates the opening of the form is tricky, ask a question about it separately.
Q:this works for sliding out a panel but how to slide in a panel.
A:You can do it by changing fourth parameter in calling Util.Animate method which is movement angle, such as: Util.Animate(button1, Util.Effect.Slide, 150, 360);
Approach #2
If you are using .NET 4 (if not replace Task with Thread), a function similar to this could be a start:
private void slideToDestination(Control destination, Control control, int delay, Action onFinish){
new Task(() =>{
int directionX = destination.Left > control.Left ? 1 : -1;
int directionY = destination.Bottom > control.Top ? 1 : -1;
while (control.Left != destination.Left || control.Top != destination.Bottom){
try{
if (control.Left != destination.Left){
this.Invoke((Action)delegate(){
control.Left += directionX;
});
}
if (control.Top != destination.Bottom){
this.Invoke((Action)delegate(){
control.Top += directionY;
});
}
Thread.Sleep(delay);
}catch{
// form could be disposed
break;
}
}
if (onFinish != null) onFinish();
}).Start();
}
Usage:
slideToDestination(sender as Control, panel1, 10, () => MessageBox.Show("Done!"));
slideToDestination(sender as Control, panel1, 0, null);
As action you would send some boolean variable to set to true so that you know that the animation has finished or some code to run after it. Beware of deadlocks when calling with a null action. You could run two animations on the same control in two different directions with the same speed, and it will stay where it was forever and of course two animations simultaneusly can make the control go infinitely in some direction because the while will never finish :)
Marino imi
Approach #3 SO: Sliding in Winforms form
...I'm making a form at the bottom of the screen and I want it to slide upwards...
but the code just runs through and shows the end position without showing the form sliding in which is what I want. I've tried Refresh, DoEvents
A>>Run the code in a background thread. Example:
int destinationX = (Screen.PrimaryScreen.WorkingArea.Width / 2) - (this.Width / 2);
int destinationY = Screen.PrimaryScreen.WorkingArea.Height - this.Height;
Point newLocation = new Point(destinationX, destinationY + this.Height);
new Thread(new ThreadStart(() =>{
do{
// this line needs to be executed in the UI thread, hence we use Invoke
this.Invoke(new Action(() => { this.Location = newLocation; }));
newLocation = new Point(destinationX, newLocation.Y - 1);
Thread.Sleep(100);
}
while (newLocation != new Point(destinationX, destinationY));
})).Start();
colSpan for DataGridViews https://www.codeproject.com/articles/34037/datagridvewtextboxcell-with-span-behaviour
https://stackoverflow.com/questions/17524739/c-sharp-datagridview-colspan
https://stackoverflow.com/questions/6936417/colspan-gridview-rows
https://social.msdn.microsoft.com/Forums/vstudio/en-US/da1b7fed-3577-4db9-a95d-b5993c4d18d5/how-to-make-colspan-in-datagridview-
https://www.aspsnippets.com/Articles/Add-Colspan-ColumnSpan-to-GridView-in-ASPNet.aspx
https://www.c-sharpcorner.com/forums/colspan-and-row-span-on-datagridview
http://www.windows-tech.info/3/89ca73fd05c73500.php
C# code snippets for DataGridView control ->
Example setting various DataGridView col properties incl tooltips
using System.Windows.Forms;
using System;
using System.Drawing;
public class DataGridViewColumnDemo : Form{
#region "set up form"
public DataGridViewColumnDemo() {
InitializeComponent();
AddButton(Button1, "Reset",
new EventHandler(ResetToDisorder));
AddButton(Button2, "Change Column 3 Header",
new EventHandler(ChangeColumn3Header));
AddButton(Button3, "Change Meatloaf Recipe",
new EventHandler(ChangeMeatloafRecipe));
AddAdditionalButtons();
InitializeDataGridView();
}
DataGridView dataGridView;
Button Button1 = new Button();
Button Button2 = new Button();
Button Button3 = new Button();
Button Button4 = new Button();
Button Button5 = new Button();
Button Button6 = new Button();
Button Button7 = new Button();
Button Button8 = new Button();
Button Button9 = new Button();
Button Button10 = new Button();
FlowLayoutPanel FlowLayoutPanel1 = new FlowLayoutPanel();
private void InitializeComponent() {
FlowLayoutPanel1.Location = new Point(454, 0);
FlowLayoutPanel1.AutoSize = true;
FlowLayoutPanel1.FlowDirection = FlowDirection.TopDown;
FlowLayoutPanel1.Name = "flowlayoutpanel";
ClientSize = new System.Drawing.Size(614, 360);
Controls.Add(this.FlowLayoutPanel1);
Text = this.GetType().Name;
AutoSize = true;
}
#endregion
#region "set up DataGridView"
private string thirdColumnHeader = "Main Ingredients";
private string boringMeatloaf = "ground beef";
private string boringMeatloafRanking = "*";
private bool boringRecipe;
private bool shortMode;
private void InitializeDataGridView() {
dataGridView = new System.Windows.Forms.DataGridView();
Controls.Add(dataGridView);
dataGridView.Size = new Size(300, 200);
// Create an unbound DataGridView by declaring a
// column count.
dataGridView.ColumnCount = 4;
AdjustDataGridViewSizing();
// Set the column header style.
DataGridViewCellStyle columnHeaderStyle =
new DataGridViewCellStyle();
columnHeaderStyle.BackColor = Color.Aqua;
columnHeaderStyle.Font =
new Font("Verdana", 10, FontStyle.Bold);
dataGridView.ColumnHeadersDefaultCellStyle =
columnHeaderStyle;
// Set the column header names.
dataGridView.Columns[0].Name = "Recipe";
dataGridView.Columns[1].Name = "Category";
dataGridView.Columns[2].Name = thirdColumnHeader;
dataGridView.Columns[3].Name = "Rating";
PostColumnCreation();
// Populate the rows.
string[] row1 = new string[]{"Meatloaf",
"Main Dish", boringMeatloaf, boringMeatloafRanking};
string[] row2 = new string[]{"Key Lime Pie",
"Dessert", "lime juice, evaporated milk", "****"};
string[] row3 = new string[]{"Orange-Salsa Pork Chops",
"Main Dish", "pork chops, salsa, orange juice", "****"};
string[] row4 = new string[]{"Black Bean and Rice Salad",
"Salad", "black beans, brown rice", "****"};
string[] row5 = new string[]{"Chocolate Cheesecake",
"Dessert", "cream cheese", "***"};
string[] row6 = new string[]{"Black Bean Dip", "Appetizer",
"black beans, sour cream", "***"};
object[] rows = new object[] { row1, row2, row3, row4, row5, row6 };
foreach (string[] rowArray in rows) {
dataGridView.Rows.Add(rowArray);
}
shortMode = false;
boringRecipe = true;
}
private void AddButton(Button button, string buttonLabel, EventHandler handler) {
FlowLayoutPanel1.Controls.Add(button);
button.TabIndex = FlowLayoutPanel1.Controls.Count;
button.Text = buttonLabel;
button.AutoSize = true;
button.Click += handler;
}
private void ResetToDisorder(object sender, System.EventArgs e) {
Controls.Remove(dataGridView);
dataGridView.Dispose();
InitializeDataGridView();
}
private void ChangeColumn3Header(object sender,
System.EventArgs e) {
Toggle(ref shortMode);
if (shortMode)
{ dataGridView.Columns[2].HeaderText = "S"; }
else
{ dataGridView.Columns[2].HeaderText = thirdColumnHeader; }
}
private static void Toggle(ref bool toggleThis) {
toggleThis = !toggleThis;
}
private void ChangeMeatloafRecipe(object sender, System.EventArgs e)
{
Toggle(ref boringRecipe);
if (boringRecipe) {
SetMeatloaf(boringMeatloaf, boringMeatloafRanking);
} else {
string greatMeatloafRecipe =
"1 lb. lean ground beef, " +
"1/2 cup bread crumbs, 1/4 cup ketchup," +
"1/3 tsp onion powder, " +
"1 clove of garlic, 1/2 pack onion soup mix " +
" dash of your favorite BBQ Sauce";
SetMeatloaf(greatMeatloafRecipe, "***");
}
}
private void SetMeatloaf(string recipe, string rating) {
dataGridView.Rows[0].Cells[2].Value = recipe;
dataGridView.Rows[0].Cells[3].Value = rating;
}
#endregion
#region "demonstration code"
private void PostColumnCreation() {
AddContextLabel();
AddCriteriaLabel();
CustomizeCellsInThirdColumn();
AddContextMenu();
SetDefaultCellInFirstColumn();
ToolTips();
dataGridView.CellMouseEnter +=
dataGridView_CellMouseEnter;
dataGridView.AutoSizeColumnModeChanged +=
dataGridView_AutoSizeColumnModeChanged;
}
private string criteriaLabel = "Column 3 sizing criteria: ";
private void AddCriteriaLabel() {
AddLabelToPanelIfNotAlreadyThere(criteriaLabel,
criteriaLabel +
dataGridView.Columns[2].AutoSizeMode.ToString() +
".");
}
private void AddContextLabel() {
string labelName = "label";
AddLabelToPanelIfNotAlreadyThere(labelName,
"Use shortcut menu to change cell color.");
}
private void AddLabelToPanelIfNotAlreadyThere(
string labelName, string labelText) {
Label label;
if (FlowLayoutPanel1.Controls[labelName] == null) {
label = new Label();
label.AutoSize = true;
label.Name = labelName;
label.BackColor = Color.Bisque;
FlowLayoutPanel1.Controls.Add(label);
} else {
label = (Label)FlowLayoutPanel1.Controls[labelName];
}
label.Text = labelText;
}
private void CustomizeCellsInThirdColumn() {
int thirdColumn = 2;
DataGridViewColumn column =
dataGridView.Columns[thirdColumn];
DataGridViewCell cell = new DataGridViewTextBoxCell();
cell.Style.BackColor = Color.Wheat;
column.CellTemplate = cell;
}
ToolStripMenuItem toolStripItem1 = new ToolStripMenuItem();
private void AddContextMenu() {
toolStripItem1.Text = "Redden";
toolStripItem1.Click += new EventHandler(toolStripItem1_Click);
ContextMenuStrip strip = new ContextMenuStrip();
foreach (DataGridViewColumn column in dataGridView.Columns) {
column.ContextMenuStrip = strip;
column.ContextMenuStrip.Items.Add(toolStripItem1);
}
}
private DataGridViewCellEventArgs mouseLocation;
// Change the cell's color.
private void toolStripItem1_Click(object sender, EventArgs args) {
dataGridView.Rows[mouseLocation.RowIndex]
.Cells[mouseLocation.ColumnIndex].Style.BackColor
= Color.Red;
}
// Deal with hovering over a cell.
private void dataGridView_CellMouseEnter(object sender,
DataGridViewCellEventArgs location) {
mouseLocation = location;
}
private void SetDefaultCellInFirstColumn() {
DataGridViewColumn firstColumn = dataGridView.Columns[0];
DataGridViewCellStyle cellStyle =
new DataGridViewCellStyle();
cellStyle.BackColor = Color.Thistle;
firstColumn.DefaultCellStyle = cellStyle;
}
private void ToolTips() {
DataGridViewColumn firstColumn = dataGridView.Columns[0];
DataGridViewColumn thirdColumn = dataGridView.Columns[2];
firstColumn.ToolTipText =
"This column uses a default cell.";
thirdColumn.ToolTipText =
"This column uses a template cell." +
" Style changes to one cell apply to all cells.";
}
private void AddAdditionalButtons() {
AddButton(Button4, "Set Minimum Width of Column Two",
new EventHandler(Button4_Click));
AddButton(Button5, "Set Width of Column One",
new EventHandler(Button5_Click));
AddButton(Button6, "Autosize Third Column",
new EventHandler(Button6_Click));
AddButton(Button7, "Add Thick Vertical Edge",
new EventHandler(Button7_Click));
AddButton(Button8, "Style and Number Columns",
new EventHandler(Button8_Click));
AddButton(Button9, "Change Column Header Text",
new EventHandler(Button9_Click));
AddButton(Button10, "Swap First and Last Columns",
new EventHandler(Button10_Click));
}
private void AdjustDataGridViewSizing() {
dataGridView.ColumnHeadersHeightSizeMode =
DataGridViewColumnHeadersHeightSizeMode.AutoSize;
}
//Set the minimum width.
private void Button4_Click(object sender, System.EventArgs e) {
DataGridViewColumn column = dataGridView.Columns[1];
column.MinimumWidth = 40;
}
// Set the width.
private void Button5_Click(object sender, System.EventArgs e) {
DataGridViewColumn column = dataGridView.Columns[0];
column.Width = 60;
}
// AutoSize the third column.
private void Button6_Click(object sender, System.EventArgs e) {
DataGridViewColumn column = dataGridView.Columns[2];
column.AutoSizeMode = DataGridViewAutoSizeColumnMode.DisplayedCells;
}
// Set the vertical edge.
private void Button7_Click(object sender, System.EventArgs e) {
int thirdColumn = 2;
DataGridViewColumn column =
dataGridView.Columns[thirdColumn];
column.DividerWidth = 10;
}
// Style and number columns.
private void Button8_Click(object sender, EventArgs args) {
DataGridViewCellStyle style = new DataGridViewCellStyle();
style.Alignment = DataGridViewContentAlignment.MiddleCenter;
style.ForeColor = Color.IndianRed;
style.BackColor = Color.Ivory;
foreach (DataGridViewColumn column in dataGridView.Columns) {
column.HeaderCell.Value = column.Index.ToString();
column.HeaderCell.Style = style;
}
}
// Change the text in the column header.
private void Button9_Click(object sender, EventArgs args) {
foreach (DataGridViewColumn column in dataGridView.Columns) {
column.HeaderText = String.Concat("Column ",
column.Index.ToString());
}
}
// Swap the last column with the first.
private void Button10_Click(object sender, EventArgs args) {
DataGridViewColumnCollection columnCollection = dataGridView.Columns;
DataGridViewColumn firstVisibleColumn =
columnCollection.GetFirstColumn(DataGridViewElementStates.Visible);
DataGridViewColumn lastVisibleColumn =
columnCollection.GetLastColumn(
DataGridViewElementStates.Visible, DataGridViewElementStates.None);
int firstColumn_sIndex = firstVisibleColumn.DisplayIndex;
firstVisibleColumn.DisplayIndex = lastVisibleColumn.DisplayIndex;
lastVisibleColumn.DisplayIndex = firstColumn_sIndex;
}
// Updated the criteria label.
private void dataGridView_AutoSizeColumnModeChanged(object sender, DataGridViewAutoSizeColumnModeEventArgs args) {
args.Column.DataGridView.Parent.
Controls["flowlayoutpanel"].Controls[criteriaLabel].
Text = criteriaLabel
+ args.Column.AutoSizeMode.ToString();
}
#endregion
[STAThreadAttribute()]
public static void Main() {
Application.Run(new DataGridViewColumnDemo());
}
}
Example for CellPainting
In the following code example, you will paint all the cells in a ContactName column using the DataGridView control's color scheme. Each cell's text content is painted in Crimson, and an inset rectangle is drawn in the same color as the DataGridView control's GridColor property.
private void dataGridView1_CellPainting(object sender,
System.Windows.Forms.DataGridViewCellPaintingEventArgs e){
if (this.dataGridView1.Columns["ContactName"].Index ==
e.ColumnIndex && e.RowIndex >= 0) {
Rectangle newRect = new Rectangle(e.CellBounds.X + 1,
e.CellBounds.Y + 1, e.CellBounds.Width - 4,
e.CellBounds.Height - 4);
using (
Brush gridBrush = new SolidBrush(this.dataGridView1.GridColor),
backColorBrush = new SolidBrush(e.CellStyle.BackColor)) {
using (Pen gridLinePen = new Pen(gridBrush)) {
// Erase the cell.
e.Graphics.FillRectangle(backColorBrush, e.CellBounds);
// Draw the grid lines (only the right and bottom lines;
// DataGridView takes care of the others).
e.Graphics.DrawLine(gridLinePen, e.CellBounds.Left,
e.CellBounds.Bottom - 1, e.CellBounds.Right - 1,
e.CellBounds.Bottom - 1);
e.Graphics.DrawLine(gridLinePen, e.CellBounds.Right - 1,
e.CellBounds.Top, e.CellBounds.Right - 1,
e.CellBounds.Bottom);
// Draw the inset highlight box.
e.Graphics.DrawRectangle(Pens.Blue, newRect);
// Draw the text content of the cell, ignoring alignment.
if (e.Value != null) {
e.Graphics.DrawString((String)e.Value, e.CellStyle.Font,
Brushes.Crimson, e.CellBounds.X + 2,
e.CellBounds.Y + 2, StringFormat.GenericDefault);
}
e.Handled = true;
}
}
}
}
MSFT Docs for Customizing the WinForms DataGridView
How to: Customize the Appearance of Cells in the Windows Forms DataGridView Control
How to: Customize the Appearance of Rows in the Windows Forms DataGridView Control
How to: Customize Cells and Columns in the Windows Forms DataGridView Control by Extending Their Behavior and Appearance
How to: Host Controls in Windows Forms DataGridView Cells
(many snippets incl: TableLayout, Drawing, GridStrip...
The GridStrip control is a custom ToolStrip that arranges
its ToolStripButton controls in a grid layout.)
The repo also has wpf docs
Enhancing Windows Forms Applications
Overriding MenuStrip class's renderer to use custom colors (shd work for all ctrls)
Custom Ttip to display img on hover
Using 'flat style' ->
...It's enough to set FlatStyle = System.Windows.Forms.FlatStyle.Flat; then FlatAppearance.BorderSize = 0; no nd to override paint...
..."Flat" controls are long-deprecated. They have been drawn the same as 3D controls ever since the introduction of Visual Styles (Windows XP). The WinForms framework has to simulate it by owner-drawing. The result is ugly, and inconsistent with the standard platform UI. The best solution is to set all controls' FlatStyle property to System....
...removing hover backgrnd ->
>>this.btnClose.FlatAppearance.MouseOverBackColor = System.Drawing.Color.Transparent;
Using the Anchor & Dock Properties
TableLayoutPanel with all childPanels' DockStyle=Fill (scroll dn to see img)
Adding label to Panel w/o dockStyleFill automatically puts it topLeft.
How to center a ctrl w/in a container?
use TableLayoutPanel w/1row&1col; lbl w/Anchor=None will alw center it vert/horiz.
Q:FlowLayoutPanel subPanels wont fill when resizing form, when the flowlayoutpanel gets bigger
A:The FlowLayoutPanel implies a Column (determined by the widest object) and a Row (determined by the tallest object) when child controls are added. This model defines the flow behavior. When you anchor or dock a child Control, this Cell (Column, Row) is the virtual container where the anchors are applied to. You may want to use a standard Panel container or TableLayoutPanel.
TableLayoutPanel auto-size behav:
>>If you set AutoSize=true & controls w/SizeType=Percent; no row/col will be clipped
TableLayoutPanel best practices >>'avoid nesting'
Q:How to design a fluid layout?
>>use one panel which docks to the left and another one which docks as "Full". This has the effect that the left part won't get wider but the right one does. This is a nifty solution for a complicated form with many controls in a fixed layout to the left and several growable grids or preview panes to the right.
>>You could put Your controls into tableLayout, and set the Dock property to fill.
ContainerControl.PerformAutoScale Method (System.Windows.Forms)
Performs scaling of the container control ****and its children****.
The following table lists cipher algorithms available in the SunJCE provider.
Algorithm Name MODES Paddings
PBEWithMD5AndDES, CBC PKCS5Padding
PBEWithMD5AndTripleDES Footnote 1,
PBEWithSHA1AndDESede,
PBEWithSHA1AndRC2_40,
PBEWithSHA1AndRC2_128,
PBEWithSHA1AndRC4_40,
PBEWithSHA1AndRC4_128,
PBEWithHmacSHA1AndAES_128,
PBEWithHmacSHA224AndAES_128,
PBEWithHmacSHA256AndAES_128,
PBEWithHmacSHA384AndAES_128,
PBEWithHmacSHA512AndAES_128,
PBEWithHmacSHA1AndAES_256,
PBEWithHmacSHA224AndAES_256,
PBEWithHmacSHA256AndAES_256,
PBEWithHmacSHA384AndAES_256,
PBEWithHmacSHA512AndAES_256
Footnote 1: PBEWithMD5AndTripleDES is a proprietary algorithm that has not been standardized.
jdk8 pbe s2 test
illustrating a proprietory cypher build: (PBEWithMD5AndTripleDESCipher )
http://hg.openjdk.java.net/jdk8/jdk8/jdk/raw-file/519f4c9ebf8d/src/share/classes/com/sun/crypto/provider/PBEWithMD5AndTripleDESCipher.java
frm 'Hacking Multifactor Authentication', Roger Grimes/Wiley (2019)
MFA vendors: xl sheet: wiley.com/go/hackingmultifactor
Open Source Intell: github.com/jivoi/awesome-osint
Pwd Recovery Qns
in 2008, Palin's YHOO act hacked in 45 min
washingtontimes.com/news/2008/sep/19/hacker_wanted_to_derail_palin
GOOG paper www.a51.nl/sites/default/files/pdf/43783.pdf
- Some recov Qns can be guessed 1st try 20% of the time
- 40% of users unable to recall their own answers
- 16% answers found in soc media profiles
Some (dated) refs from the paper above -> (need to chk)
Secrets, Lies, and Account Recovery:
Lessons from the Use of Personal Knowledge Questions at Google
Mike Just. Designing and Evaluating Challenge-Question
Systems. IEEE Security & Privacy Magazine, 2(5), 2004.
[21] Mike Just and David Aspinall. Personal Choice and Challenge
Questions: A Security and Usability Assessment. SOUPS 09:
The 5th Symposium on Usable Privacy and Security, 2009.
Rachael Pond, John Podd, Julie Bunnell, and Ron Henderson.
Word Association Computer Passwords: The Effect of
Formulation Techniques on Recall and Guessing Rates.
Computers & Security, 2000.
Ariel Rabkin. Personal knowledge questions for fallback
authentication: Security questions in the era of Facebook.
SOUPS 08: The 4th Symposium on Usable Privacy and
Security, 2008
Stuart Schechter, A. J. Bernheim Brush, and Serge Egelman.
Its No Secret: Measuring the security and reliability of
authentication via secret questions. 2009 IEEE Symposium
on Security and Privacy, 2009.
Moshe Zviran and William J. Haga. A Comparison of
Password Techniques for Multilevel Authentication
Mechanisms. Computer Journal, 36(3):227237, 1993.
Adobe color schemes HTML / CSS Stuff
https://css-tricks.com/guides/
https://codepen.io/
https://developer.mozilla.org
Using the <abbr> tag
The following text uses the abbr tag; which shows a tooltip automatically : TLA
Using the <dl>, <dt> and <dd> tags
Term: <dl> tag
Definition: The <dl> tag is used to code a description list consisting
of description terms alternating with groups of one or more detail descriptions.
Term: <dt> tag
Definition: The <dt> tag is used to code a description term
in a description list, leading into one or more detail description elements.
Term: <dd> tag
Definition: The <dd> detail description tag is used to code the
description of an item inside a dl element for a description list.
Using the <mark> tag
You searched for "url link"
1 result:
<a> tag example:
<a href="target url">link text</a>
document.getElementById('cloud-tag-container').oncontextmenu = function(evt) {
if (evt.target.classList.contains('tag')) {
// then you right clicked on the span with class 'tag'
// perform your actions here
}
}
(clouds):
You could easily do it with your CMS and css
div class=".cloud"
a href="#" class="weight-1"Cloud/a
a href="#" class="weight-2"Cloud/a
/div
css:
.cloud .weight-1 { font-size: 10px; }
.cloud .weight-2 { font-size: 15px; }
Your CMS would generate the weights and apply the appropriate styling.
var backgrounds = ['gray', 'red', 'blue'];
var elements = document.getElementsByClassName('blogpost');
for (i=0; i [ elements.length; i++) {
// the following could be slightly optimized by storing the value to
// backgrounds.length before the loop starts
elements[i].className += ' ' + backgrounds[i%backgrounds.length];
Ref: dojo.lang.hitch:
Pitfall: Functions Inside Methods Shadow thisYou often nest function definitions in JavaScript, because functions can be parameters (e.g., callbacks) and because they can be created in place, via function expressions. This poses a problem when a method contains a normal function and you want to access the former's this inside the latter, because the method's this is shadowed by the normal function's this (which doesn't even have any use for its own this).
In the following example, the function at (1) tries to access the method's this at (2):
var obj = { name: 'Jane', friends: [ 'Tarzan', 'Cheeta' ],
loop:
function () {
'use strict';
this.friends.forEach(
function (friend) { // (1)
console.log(this.name+' knows '+friend); // (2)
}
);
}};
This fails, because the function at (1) has its own this, which is undefined here
:> obj.loop();TypeError: Cannot read property 'name' of undefined
There are three ways to work around this problem.
Workaround 1: that = thisWe assign this to a variable that won't be shadowed inside the nested function:loop: function () { 'use strict'; var that = this; this.friends.forEach(function (friend) { console.log(that.name+' knows '+friend); });}Here's the interaction:> obj.loop();Jane knows TarzanJane knows Cheeta
Workaround 2: bind()We can use bind() to give the callback a fixed value for thisnamely, the method's this (line (1)):loop: function () { 'use strict'; this.friends.forEach(function (friend) { console.log(this.name+' knows '+friend); }.bind(this)); // (1)}
Workaround 3: a thisValue for forEach()
A workaround that is specific to forEach() (see Examination Methods) is to provide a secondparameter after the callback that becomes the this of the callback:
loop:
function () { 'use strict';
this.friends.forEach(function (friend) {
console.log(this.name+' knows '+friend);
}, this);}
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
an antipattern used by programmers who want to access a property whose name is stored in a variable propName: var value = eval('obj.'+propName);The idea makes sense: the dot operator only supports fixed, statically provided property keys. In this case, the property key is only known at runtime, which is why eval() is needed in order to use that operator. Luckily, JavaScript also has the bracket operator, which does accept dynamic property keys. Therefore, the following is a better version of the preceding code: var value = obj[propName];
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Warning
Avoid invoking hasOwnProperty() directly on an object, as it may be overridden (e.g., by an own property whose key is hasOwnProperty): > var obj = { hasOwnProperty: 1, foo: 2 }; > obj.hasOwnProperty('foo') // unsafe TypeError: Property 'hasOwnProperty' is not a functionInstead, it is better to call it generically (see Generic Methods: Borrowing Methods from Prototypes): > Object.prototype.hasOwnProperty.call(obj, 'foo') // safe true > {}.hasOwnProperty.call(obj, 'foo') // shorter true
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Check for the existence of a property via in and hasOwnProperty // All properties: if (obj.foo) // no if (obj.foo !== undefined) // no if ('foo' in obj) ... // yes // Own properties: if (obj.hasOwnProperty('foo')) ... // risky for arbitrary objects if (Object.prototype.hasOwnProperty.call(obj, 'foo')) ... // safe
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
They can be used in the following widgets: AccordionContainer, Menu, TabContainer, TitlePane, Tree, and all button widgets.
for claro; add: media="screen" for JS ; add: data-dojo-config="isDebug: 1, async: 1, parseOnLoad: 1"
http://svn.dojotoolkit.org/src/dojox/trunk/storage/README
http://svn.dojotoolkit.org/src/dojox/trunk/storage/CookieStorageProvider.js
http://svn.dojotoolkit.org/src/dojox/trunk/storage/LocalStorageProvider.js
(reddit) Using localstorage extensively any risks
Dojo Grid options: gridX & link to its playground dGrid
dojo.registerModulePath = function(/*String*/moduleName, /*String*/prefix){
// summary:
// Maps a module name to a path
// description:
// An unregistered module is given the default path of ../[module],
// relative to Dojo root. For example, module acme is mapped to
// ../acme. If you want to use a different module name, use
// dojo.registerModulePath.
// example:
// If your dojo.js is located at this location in the web root:
// | /myapp/js/dojo/dojo/dojo.js
// and your modules are located at:
// | /myapp/js/foo/bar.js
// | /myapp/js/foo/baz.js
// | /myapp/js/foo/thud/xyzzy.js
// Your application can tell Dojo to locate the "foo" namespace by calling:
// | dojo.registerModulePath("foo", "../../foo");
// At which point you can then use dojo.require() to load the
// modules (assuming they provide() the same things which are
// required). The full code might be:
// | {script type="text/javascript"
// | src="/myapp/js/dojo/dojo/dojo.js">{/script}
// | {script type="text/javascript"]
// | dojo.registerModulePath("foo", "../../foo");
// | dojo.require("foo.bar");
// | dojo.require("foo.baz");
// | dojo.require("foo.thud.xyzzy");
// | {/script]
var paths = {};
paths[moduleName.replace(/\./g, "/")] = prefix;
require({paths:paths});
};
replace: function(tmpl, map, pattern){
// summary:
// Performs parameterized substitutions on a string. Throws an
// exception if any parameter is unmatched.
geom.setContentSize = function setContentSize(/*DomNode*/ node, /*Object*/ box, /*Object*/ computedStyle){
// summary:
// Sets the size of the node's contents, irrespective of margins,
// padding, or borders.
// node: DOMNode
// box: Object
// hash with optional "w", and "h" properties for "width", and "height"
// respectively. All specified properties should have numeric values in whole pixels.
// computedStyle: Object?
// This parameter accepts computed styles object.
// If this parameter is omitted, the functions will call
// dojo/dom-style.getComputedStyle to get one. It is a better way, calling
// dojo/dom-style.getComputedStyle once, and then pass the reference to this
// computedStyle parameter. Wherever possible, reuse the returned
// object of dojo/dom-style.getComputedStyle().
// | require(["dojo/dom-style"], function(domStyle){
// | domStyle.set("thinger", "opacity", 0.5); // == 0.5
// | });
dojo.style = function(node, name, value){
// summary:
// Accesses styles on a node. If 2 arguments are
// passed, acts as a getter. If 3 arguments are passed, acts
// as a setter.
dojo.style("thinger", "opacity", 0.5); // == 0.5
| dojo.style("thinger", {
// | "opacity": 0.5,
// | "border": "3px solid black",
// | "height": "300px"
// | });
Demo: jsonp Dojo pull requests
This page has been updated with a live JSONP pull list
I have an app whose code is written in TypeScript and that uses jQuery to attach functions to page events and it works pretty well; although, like you, I found it to obe a struggle to identify the magic formula to make it all work.
My TS file starts with a reference to the jQuery type file:
/// [reference path="../typings/globals/jQuery/index" /]
then there's all of my app-specific code that I will omit here. Finally at the end, I have my jQuery code to hook everything up:>
$(document).ready(function () { $("#answerRequest").click(function (e) { e.preventDefault(); let answer = chatty.ask($("#question").val()); $("#answer").val(answer); }); });
That should all be pretty familiar. The chatty object is instantiated elsewhere.
Now that I have my .ts file, I compile it with tsc, then I use browserify to turn the .js from the tsc output into JS that will run happily in the browser.
In the HTML, I include a script tag that points to the output of the browserify step, and it works just as if I had written the whole thing in JS to begin with.
Console output...
NYC: The place that put US cheeses on the map
"one of Manhattans premier cheese shops" in Chelsea Market
Herbal Fruit Jams http://salliesgreatest.com
Fairy Tale Brownies (brownies.com) [[Has a good-looking coll of blondies too.]]
Blue Martini Companies -> Taubman Centers, Michigan CEO Mark Vasu (another link to someone Mitsu calls "Stinker Pinker")
Food:
Seasonality table Seasonal food calendar (note: this site requires you to enter a New York zip code. 10003 is one that will work)
fallingfruit.org source for trail berries
Online Pie Order Sources
Featured on Good Morning America and a National Pie Championship winner, the Michigan 4-Berry Pie is Achatz's signature dessert!
Chocolate Pecan. VOTED DC'S BEST PIE SEVEN YEARS RUNNING!
Sweet-n-salty Must check other choices; also has a "pie club" 6 month plan (cld be same as above site?)
MISSISSIPPI MUD PIE imported Belgian chocolate, brownie streusel and a chocolate glaze in a chocolate cookie crust. (Also has key lime etc.)
They have a HUGE selection
This is our famous pecan pie, but in a fudgy deep dish form. It's packed full of delicious, Southern pecan halves-a full 2 pounds of nuts.
Not overly sweet, this pie is a Two Fat Cats favorite. 7?/9? available
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et
dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex
ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
mollit anim id est laborum.
https://gutenberg.org/ebooks/1056 (Martin Eden, Jack London)
https://en.wikipedia.org/wiki/The_Strange_Case_of_Peter_the_Lett (Simenon's 1st Maigret)
https://en.wikipedia.org/wiki/The_Resistible_Rise_of_Arturo_Ui (Brecht)
https://en.wikipedia.org/wiki/%C3%89crits (Lacan)
https://en.wikipedia.org/wiki/Manhattan_Transfer_(Dos Passos)
https://en.wikipedia.org/wiki/The_General_of_the_Dead_Army_(Ismael Kadare)
https://en.wikipedia.org/wiki/The_Notebooks_of_Malte_Laurids_Brigge (Rilke's only noveL)
https://en.wikipedia.org/wiki/Tristes_Tropiques
https://en.wikipedia.org/wiki/The_Tartar_Steppe
https://en.wikipedia.org/wiki/Confusion_of_Feelings
https://en.wikipedia.org/wiki/Bonjour_Tristesse
https://en.wikipedia.org/wiki/The_Lost_Honour_of_Katharina_Blum
The Association of Small Bombs By Karan Mahajan
A finalist for the National Book Award, Mahajan's novel smart, devastating and unpredictable opens with a Kashmirian terrorist attack in a Delhi market, then follows the lives of those affected. This includes Deepa and Vikas Khurana, whose young sons were killed, and the boys' injured friend Mansoor, who grows up to flirt with a form of political terrorism himself. As the narrative suggests, nothing recovers from a bomb: not our humanity, not our politics, not even our faith.
Review: http://www.nytimes.com/2016/03/20/books/review/the-association-of-small-bombs-by-karan-mahajan.html
Dark Money: The Hidden History of the Billionaires Behind the Rise of the Radical RightBy Jane MayerIn 1980 Charles and David Koch decided they would spend vast amounts of their fortune to elect conservatives to all levels of government, and the world of American politics has never been the same. Mayer spent five years looking into the Koch brothers' activities, and the result is this thoroughly investigated, well-documented book. It cannot have been easy to uncover the workings of so secretive an operation, but Mayer has come as close to doing it as anyone is likely to anytime soon.
Review: http://www.nytimes.com/2016/01/24/books/review/dark-money-by-jane-mayer.html
S Korean films All's not quiet on the eastern front - Film
Times, The (London, England) - October 1, 2005
Author/Byline: Ian JohnsSection: FeaturesPage: London Film Festival 26Readability: >12 grade level (Lexile: 1440)
South Korea is the newest pretender to the world-cinema throne, and it has the energy and diversity to back up its claim, finds Ian Johns
Every ten years or so a new country seems to take the lead in ground-breaking cinema. After the Second World War it was the Italian neo-realists. In the 1960s it was the French New Wave. More recently, in the 1990s, China led the field before being edged out by Iran. Now South Korea has seized the baton with a group of eclectic, provocative film-makers producing a diverse range of work, a taste of which can be experienced in this year's festival.
The films include Kim Dae-Seung's Blood Rain, in which an imperial investigator in the early 1800s is faced with a series of grisly murders on an island. It's a labyrinthine mystery, full of evocative period detail, involving a paper-mill-owning family, a reclusive young swordsman and sins of the past.
Another mystery, Wang Cheol-Mean's Spying Cam, couldn't be more different as political and sexual barriers break down between two men as they tape themselves trapped in a motel room.
A rising directors making an impact overseas is Park Chan-Wook. His festival film Sympathy for Lady Vengeance is a funny, violent story about a murderess pursuing a vendetta that should provide a slightly less daunting experience for mainstream audiences than his gruesome but gripping cult hit Oldboy. Just as the hero of that film was abducted and detained for 15 years without knowing the reason why, the heroine of Lady Vengeance serves 13 years in jail on false charges of kidnapping and infanticide.
"I'm often misunderstood as a director of violence," Park has said, "but really I want to show how violence makes the perpetrator and the victim destroy themselves." And always wrapped up in his fascination with violence, vengeance and taboos is an interest in ethical questions.
Park's high energy, provocative film-making reflects an erupting energy among Korean film-makers. Since South Korea's military dictatorship ended in 1992 with the election of a civilian president, there has been a breaking down of censorship barriers and a rapid growth in production, helped by a quota system that ensures a high proportion of screen time for local films.
Park describes himself as part of "a generation thirsty for more cinematic expression and the richness and diversity of cinema". According to Anthony Leong, the author of Korean Cinema: The New Hong Kong, "Korean cinema is a melting pot of different types of film-making". He cites many of the film-makers now making an impact as being young, Western-educated and fans of Western, Hong Kong and Japanese cinema, as well as the French New Wave. "They take all these techniques, perspectives and genres and meld them into something unique."
Kim Ki-Duk has a reputation as one of the bad boys of Korean cinema. His films include Samaritan Girl, a gritty tale of teen prostitution, Bad Guy, a sordid account of a pimp's brutal recruitment of a student, and the revenge yarn 3 Iron.
Yet he also made Spring, Summer, Autumn, Winter...and Spring (2003), a quasi-Buddhist tale filled with serene natural imagery. It made less than $300,000 in Korea but became the most successful Korean film at the US box office, earning $2.3 million.
Success abroad is becoming increasingly important for the Korean film industry, which has moved beyond domestic crowd-pleasers such as cop dramas and romantic comedies into darker thrillers, martial-arts adventures, reality-based war epics and distinctive auteur offerings. Even two of South Korea's biggest mainstream hits -Silmido (about a convict commando unit brutally trained in 1968 to kill the North Korean leader) and Taegukgi (two brothers pitted against each other during the Korean war) are tackling the country's troubled past.
For Kim Dong-Ju, the head of the Seoul-based production and distribution company Show West, "Korea used to be an importing nation, now it will be an exporter". He is working with other Asian backers on such projects as Chen Kaige's The Promise, touted as "the Lord of the Rings of the East". "We are just a beginner," Kim says, "but we want to learn how to make movies like Hero and House of Flying Daggers."
At the same time the French company MK2 has invested in two of Hong Sang-Soo's recent films, Woman is the Future of Man and Tale of Cinema. The latter, a London festival screening, is an intriguing blurring of fact and fiction as a struggling film-maker bumps into a film star outside a cinema after seeing a movie that was inspired by his life. For film-makers such as Hong, whose low-key, observant style has a relaxed charm reminiscent of Eric Rohmer, foreign backing is invaluable.
It also means that first-time directors such as Lee Yoon-Ki can hope to carve out a career. His festival film This Charming Girl, shot entirely with a hand-held camera, is a gentle portrait of a shy post-office worker who summons up the courage to invite a regular customer to dinner but is stood up and later offered a lame excuse. We're left, like her, wondering whether to believe him or not.
An indication of South Korea's passion for cinema was amply demonstrated in 1999 when a change to South Korea's film quota system was suggested. There were hunger strikes and boycotts of Hollywood films -100 film professionals even shaved their heads in protest. But while Kim acknowledges that the industry now boasts many committed, talented people, he doesn't see it as guaranteeing a rosy future: "We just need to try harder to make better films."
* Blood Rain, Oct 28, NFT1 & Oct 31, Odeon West End 2; Spying Cam, Oct 21 & 24, ICA; Sympathy for Lady Vengeance, Oct 22, Odeon West End 2; Tale of Cinema Oct 30 & Nov 2, NFT1; This Charming Girl, Oct 28, NFT2, & 31, NFT1
ASIA AT THE FESTIVAL:
IF YOU WERE ME 2 Shorts by leading South Korean directors for the National Human Rights Commission.
KEKEXILI: MOUNTAIN PATROL Inspired by the true story of a near-suicidal pursuit of poachers by game wardens in Tibet.
SHANGHAI DREAMS Family saga of how factory workers relocated to remote areas in the 1960s tried to move back to the cities in the 1980s.
STOLEN LIFE Poignant Chinese story of a college graduate who gets pregnant.
Tokyo? Yo! - Japan
Sunday Times, The (London, England) - January 2, 1994
Author/Byline: Tom HineySection: FeaturesPage: 6/14Readability: >12 grade level (Lexile: 1410)
Instead of imitating Western culture, the Japanese are now influencing it.
There was a time when the one thing Britain didn't import from Japan was its culture. The electronics with which to create and replay our own sights and sounds were worth every yen, but there it stopped. With the exception of art house stalwarts and credit card hippies (Pounds 80-a-head Now and Zen dinners, Buddhism courses in South Ken), most of us presumed that Japanese culture was either a feeble imitation of the West (of which the karaoke machine was the frightening embodiment) or consisted of tea gardens and samurai.
But no more. Watch out for manga cartoons demented full-length adult animation films and for cyberpunk movies, a sort of Dr Who for grown-ups. From the burgeoning of import mixes by Tokyo club DJs, to the staying power of Japanese designers and young novelists, Britain is waking up to the joys of Japanese pop culture.
The Land of the Rising Street Cred has graduated from its bug-eyed crash course in Hollywood and Carnaby Street and is creating its own versions of the 1990s in ways both distinctly Japanese and yet with all the familiar idiom of a generation brought up on the Stones and Stanley Kubrick. Forget bullet trains and company anthems, this is the subversive imagination of a new, streetwise Japan.
You could not get further from the salarymen cliches or chocolate box Wish You Were Here images of Japan than manga cartoons. These are an adult-rated, unashamed diet of what one reviewer described as ``violent sex and sexy violence" and now, as they say, available in your local video store.
While manga has a long tradition in Japan originating in risque comic-books, the most successful of which were later filmed it has taken a generation of directors reared on Ridley Scott and David Cronenberg to take it out of crude samurai swashbuckling into the post-apocalyptic sophistication of Akira, Legend Of The Overfiend or Dominion-Tank Police. The action is gruesome, the plots generally baffling and some people here don't seem to be able to get enough.
When the ICA in London, the flagship of art-houses, showed a selection of manga animes two years ago, under the billing A Celebration Of Japanese Animation, they were besieged by young manga freaks. Long tantalised by references in their computer magazines, they were desperate to see classics such as Akira, by Katsuhiro Otomo, about a boy turned by scientists into a monster, on the big screen. All box office records were broken as hordes of people who normally wouldn't go within spitting distance of a contemporary arts institute flocked to see Venus Wars, Fist Of The North Star, or Robot Carnival. Such was the interest that Island International, part of the record company, bought distribution rights and launched Manga Video. Each title so far has gone straight into the top 10 national video charts within weeks, displacing staple bestsellers, but soon afterwards disappearing a true manifestation of cult buying.
Kenny Penman, who as the manager of the Forbidden Planet comic and video store in Edinburgh has witnessed the manga-raiders at work, says: ``It's a fanatical thing they know the release dates before we do, and always buy the video straight away. Manga is bringing out two a month now and they always sell about 12,000 in the first week."
Shinya Tsukamoto's Tetsuo a black-and-white movie about a man who turns into, yes, a monster will not be shown properly in Britain until this spring, but is already regarded as the ultimate cyberpunk movie; what more likely nation than Japan to create such a gory and compulsive vision of techno-madness. Subtitles, which are always liable to damage a film's wide popularity, are badly needed in Tetsuo: it is like a demented, hour-long pop video, and if you think the violence is disturbing, wait until you hear the soundtrack.
Japanese jazz funk has also been grabbing selected spotlights lately; Mo Wax is a new London record label concentrating on Tokyo club releases, whose compilation Jazz Hip Jazz (released last month) is genuinely ``available everywhere". According to Ashley Heath, of The Face magazine, groups such as UFO (United Future Organisation) are being taken ``very seriously" in British clubs; the birthplace of modern electronics has come into its own in the age of mixing and sampling. The chain-smoking, stubble-weary young Japanese DJs who are masterminding this surge are the epitome of scowling hip. The most sought-after soundtrack writer and musical polymath of our day is the seasoned Ryuichui Sakamoto (Last Emperor, Wild Palms), but it is Jap Jazz that has the music press excited. ``How much is a ticket to Tokyo?" finished one breathless review.
Similarly with fashion few Londoners would continue spending Tokyo prices on Issey Miyake, Yohji Yamamoto or Rei Kawakubo if these designers were just novelty weirdos or expensive copycats. They're different enough to be fresh and controversial, but not so different as to be tedious, which is really all the fashion and music worlds ask of anybody.
Even the trend value of Japanese lager, that apparently most self-generated of marketing whims, was not entirely arbitrary. These days we don't feel guilty about appreciating supermarket aesthetics, and when it comes to imaginative packaging you can't beat the Japs: if you thought Lucozade was inventive with its isotonic-in-a-bag, try a magnificent two-litre can of Sapporo (so outsize that it's got a metal handle) or the stationery departments at Muji or Sogo stores.
British publishers are meanwhile busily hunting down the cult novel the Catcher In The Rye of downtown Tokyo. Contestants include most of Shusaku Endo's books, which always get rave reviews here. Endo is set for household naming after Martin Scorsese's planned adaptation of his classic novel, Silence. His novels portray a gritty and paranoid Tokyo, as do those of the excellent Haruki Murakami. Murakami's characters, in books such as A Wild Sheep Chase or The End Of The World and the Hard Boiled Wonderland, listen to the Stones, read Sherlock Holmes and hate offices. Far from this making him unpopular in Japan, he is the country's biggest selling writer, and Penguin reports healthy sales here. Faber and Faber has its money on Banana Yoshimoto, whose first novel, Kitchen, is about ``mothers, transsexuality, bereavement, kitchens, love and tragedy", and comes out in paperback later this month.
So forget Yoko Ono, David Sylvian, Seven Samurai, misspelt Bowie sleeve notes and Dylan making a quick buck at Budokan. And certainly forget, as emblems of Japan, Nikon cameras, Nissan car factories, over-priced Impressionists, endurance game shows, and the nightmare that is karaoke. Pick up instead a copy of Super Play, a 50,000-circulation independent British computer games glossy, enjoy its fixation with Japanese snacks and toys and read its column from Tokyo, called Live From City Hell.
Elsewhere, be tolerant of the occasional lapse in Japanese bicultural hipness, such as in Banana Yoshimoto's dedication of her novel to a Mr Jiro Yoshikawa, ``who introduced me to Mike Oldfield's wonderful piece of music". You can forgive a nation for being 10 years late with punk and 20 years late with ska, but to name a whole chapter of your first novel after Moonlight Shadow is pushing it.
Never mind it won't seem to matter when you pick up the latest action-packed issue of Manga Mania magazine. Whatever happened to the computer dweeb?
Jonathan Ross presents a programme on manga on BBC2 at 7.30pm on Friday; Akira is on BBC2 at 10.30pm on Saturday.
Record: 986546909Copyright: (c) Times Newspapers Limited 1994, 2003
Startup Stuff
Startup Playbook
Startup Reading/Links
[/r/bestof] /u/divbase explains a lot of things he did wrong with his failed startup Bootstraping a SAAS
Startup Pulse Job Search
20161103mpt java job trends xpages srch results
20161103mpt salary rate java freelance consultant 2016 Yahoo srch results
20161103mpt salary rate java freelance consultant 2016 BING srch results
Yahoo & Bing job srch (1 pg ea) "ibm domino jobs contract -pizza -delivery -driver&n=100&sort=new"
Params for Job srch:
=ibm+domino+jobs+java+contract+us+-canada+-uk +-administrator+-pizza+-delivery+-driver+n=100+sort=new
Salary
"Knowing a niche technology and/or industry, finding a company that needs to staff and going in as a self-employed contractor instead of an employee.
I had friends making $200 an hour in 1999 because of this.
I'd like to say they were working in Lotus at the time, but I don't really remember."
https://news.ycombinator.com/item?id=5804798
Corina Stanis (Duke): The Art of Distances (2018)
essay on aeon: On Tact in Dark Times, 'masters of diplomacy & tact':
Theodor Adorno, US exile: Minima Moralia ('51)
Anne DuFourmantelle: Power of Gentleness: Meditations on the Risk of Living (2013)
Pierre Zaoui: Discretion, or the Art of Disappearing (2013)
Thomas Mann try to locate his essay Brother Hitler (esquire 39); ref: Nazism centered in cultural mvmts of fin de siecle era
After WWI he wrote 'Confessions of a non-politcal man', ref: 'An authoritarian state is proper & becoming to Germans'
from zauberBrg, which 'acutely diagnozed the malady afflicting central Europe': character Leo Naptha (a fanatical Jes. priest)
'The mystery and precept of our age (1924) is not liberation and the development of the ego. What our age needs, what it demands, what it will create for itself; is - terror.'
Soviet exiles in the US during the ColdWar era (ref Nabokov, cousin of Vlad, opera composer, on Shostakovich's 7th et al) - cia-backed cultural mouthpieces took the stance that 'without freedom art can't exist' etc., many voices incl Vlad Nab. put down all Soviet writers/musicians; refs from art:
20th Century political art: Brecht, Neruda, Rivera
"The highwater mark for Spanish lit. coincided w/ruthless religious autocracy": Velasquez, El Greco, Cervantes, Tomas Luis de Victoria.
Soviets: Eisenstein, Meyerhold, Akhmatova.
Tempo Rubato (it., stolen time) Singing technique by which a vocalist plays around with, but ultimately obeys, a song's timing.
The Lost Tools Of Learning By Dorothy Sayers
https://en.wikipedia.org/wiki/Katapayadi_system
https://en.wikipedia.org/wiki/Method_of_loci
10 Athleic NYC items
Loop the rez at night
Decline using yr car or Uber, walk instead!
Appreciate the facilities in midtown
Enjoy a heavenly bliss out after a run
Pick up some post-run energy with an espresso
Hit the roads on your bike and ride
Kiss the blues goodbye @ CBGB
Catch some tree-time by going outdoors
Rise above the smoke/smog and look down
Shoot some pool everyday
test Div One Content
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam dapibus efficitur gravida. Curabitur semper dignissim posuere. In augue nisl, malesuada a tellus quis, sagittis fermentum arcu. Duis at sodales erat, non finibus orci. Phasellus in auctor justo, sed congue nisl. Nulla iaculis, purus id sollicitudin hendrerit, mi orci maximus dolor, a feugiat elit justo id magna. Nulla vestibulum ligula sed accumsan finibus. Phasellus ornare eros nisl, sit amet mattis libero vulputate in. Ut a blandit leo, a rhoncus velit. Aliquam sit amet dolor nisi. Duis lobortis mauris arcu, ut auctor massa venenatis id. Cras imperdiet nibh non purus vulputate, vel lobortis neque pretium. Ut vel gravida libero. Aliquam erat volutpat.
Interdum et malesuada fames ac ante ipsum primis in faucibus. In hac habitasse platea dictumst. Suspendisse potenti. Nulla ut lacus vitae velit suscipit pulvinar in eu purus. Mauris dapibus nunc risus, ac tempor felis congue id. Integer nec placerat enim. Suspendisse lobortis fermentum ornare. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla at ligula eu mi molestie venenatis at eu ex. Nam feugiat aliquet massa, tincidunt iaculis nunc tincidunt eget. Etiam in iaculis leo. Etiam vulputate in est a aliquet. Proin fermentum felis cursus leo egestas elementum.
Suspendisse in aliquam tortor, quis elementum neque. Donec ultrices purus sed est aliquam vulputate. Donec lobortis, mi sed dapibus auctor, nunc lorem feugiat dolor, ac aliquet sem orci at nisl. Aliquam tincidunt ante leo. Sed pulvinar elit nec mi mollis, eget laoreet erat cursus. Sed euismod in est ac feugiat. Ut ac neque sit amet tortor bibendum accumsan. Pellentesque sed fringilla ipsum. Fusce velit tortor, sagittis vel velit non, lacinia auctor purus. Donec commodo sed arcu eget ultricies. Proin feugiat dignissim hendrerit. Vestibulum efficitur tincidunt rhoncus. Pellentesque lobortis justo vitae nunc bibendum, nec consectetur leo tristique.
Maecenas a eleifend ligula, ac finibus nibh. Quisque ullamcorper tortor pulvinar, pellentesque sapien sit amet, semper nunc. Praesent luctus erat in lacus efficitur, tincidunt laoreet ante suscipit. Nullam laoreet vestibulum sagittis. Suspendisse tristique, massa eget ultricies tincidunt, ipsum felis lobortis est, vitae interdum augue enim ut turpis. Quisque a augue nisi. Proin cursus ligula at pulvinar eleifend. Nunc sollicitudin ultrices mi eget varius. Etiam lectus risus, vulputate nec leo sed, ultrices gravida dolor. Donec ultricies sodales justo eu vehicula. Sed sit amet ultrices dolor. Nunc a congue sem.
test Div Two Content
1914 translation by H. Rackham
"But I must explain to you how all this mistaken idea of denouncing pleasure and praising pain was born and I will give you a complete account of the system, and expound the actual teachings of the great explorer of the truth, the master-builder of human happiness. No one rejects, dislikes, or avoids pleasure itself, because it is pleasure, but because those who do not know how to pursue pleasure rationally encounter consequences that are extremely painful. Nor again is there anyone who loves or pursues or desires to obtain pain of itself, because it is pain, but because occasionally circumstances occur in which toil and pain can procure him some great pleasure. To take a trivial example, which of us ever undertakes laborious physical exercise, except to obtain some advantage from it? But who has any right to find fault with a man who chooses to enjoy a pleasure that has no annoying consequences, or one who avoids a pain that produces no resultant pleasure?"
"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains."
Hi friends
test Div Three Content
Gist Output:
Output here...
John Milton, 1608 - 1674
When I consider how my light is spent,
Ere half my days in this dark world and wide,
And that one talent which is death to hide
Lodged with me useless, though my soul more bent
To serve therewith my Maker, and present
My true account, lest He returning chide;
"Doth God exact day-labor, light denied?"
I fondly ask. But Patience, to prevent
That murmur, soon replies, "God doth not need
Either man's work or His own gifts. Who best
Bear His mild yoke, they serve Him best. His state
Is kingly: thousands at His bidding speed,
And post o'er land and ocean without rest;
They also serve who only stand and wait."
Acquainted with the Night, ROBERT FROST
I have been one acquainted with the night.
I have walked out in rain-and back in rain.
I have outwalked the furthest city light.
I have looked down the saddest city lane.
I have passed by the watchman on his beat
And dropped my eyes, unwilling to explain.
I have stood still and stopped the sound of feet
When far away an interrupted cry
Came over houses from another street,
But not to call me back or say good-bye;
And further still at an unearthly height,
One luminary clock against the sky
Proclaimed the time was neither wrong nor right.
I have been one acquainted with the night.
//Before running, will have to compile with jsoup jar:
//export CLASSPATH= $CLASSPATH:/usr/share/java/jsoup/jsoup.jar
//------------------------------------------------------------------------------
//Before running, will have to compile with jsoup & apache commons jars:
//export CLASSPATH=$CLASSPATH:/usr/share/java/jsoup/jsoup.jar:
//usr/share/java/apache-commons-lang3.jar
//------------------------------------------------------------------------------
import java.net.*;
import java.io.*;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.apache.commons.lang3.StringEscapeUtils;
public class URLRdr {
public static String getText(String url) throws IOException {
URL website = new URL(url);
String strBody = "";
Document doc = Jsoup.connect(url).get();
Elements bodyTxt = doc.select("body");
for (Element el : bodyTxt){
System.out.println("bodyTxt: "+el);
strBody = strBody + el.toString();
}
//escape the string for JSON
strBody = StringEscapeUtils.escapeJson(strBody);
return strBody;
}
public static void main( String[] args ) throws IOException{
Other reading HN: the good parts Conversations about Software Engineering (some chkd) Harvard Biz Rev Frontline Shows Movie scripts TED Talk transcripts p.m: lake mule layer blossom pill shoe obvious current toast senior gesture until
rp: (num) YYOD + ### nr agitProp + (txt) riot-oriented hollins + (FN) iit bobby prem
ml: (col) flow'd like de river + (txt) 100EDOok + (numTxt) hollins bonMotty
g: (reg) hollins BangAcn
Jan:
rp:"Bananas x 3 (hin, FN)"; "(txt) hollins suscept. to this kalam"; "TIK acr from Prem's place";
ml:"Band's narrow? Breakout, probly (1st)";"(num) MMDDOdipti"; "mercurialRepo chinese equiv.";
fin:"(txt) hollins surpei"; "Change b/h pool 2 get (1st)"; "Huge chap w/a secretary fan (nick)"
May eom:
rp:"(num) YYOBS + ##SiteOfHammer"; "(col) ashD or orcl";"(1 wd) 1 lvl up frm advSpkrs"
ml:"(num) YYOD + ### nr agitProp"; "(col) TrikeBkCol"; "(1 wd) ei-di-ding (x2) goldee"
June:
"(col) ashD or orcl", "(num) tri cld've been flooded if in nyc", "(F wd) astr Pan had william's candy"
"(col) RMGSHouse", "(num) ##ReiSol + ###Haussm.", "(1 wd) Paresh: isi qual __ refl"
Lazy sequences implementation for Java 8
by Tomasz Nurkiewicz (of Functors/Monad fame) -> article link (a lib)
(also see: https://www.nurkiewicz.com/2013/05/lazy-sequences-in-scala-and-clojure.html)
Can be used for loading batches (pages) of data from database. Using ResultSet or Iterator is cumbersome but loading whole data set into memory often not feasible. An alternative involves loading first batch of data eagerly and then providing a function to load next batches. Data is loaded only when it's really needed and we don't suffer performance or scalability issues.
First let's define abstract API for loading batches of data from database:
public List loadPage(int offset, int max) {
//load records from offset to offset + max
}
I abstract from the technology entirely, but you get the point. Imagine that we now define LazySeq that starts from row 0 and loads next pages only when needed:
public static final int PAGE_SIZE = 5;
private LazySeq records(int from) {
return LazySeq.concat(
loadPage(from, PAGE_SIZE),
() -> records(from + PAGE_SIZE)
);
}
Neal Ford's EITHER (as a tree for pattern matching) ->
Either trees and pattern matching
https://www.ibm.com/developerworks/java/library/j-ft14/index.html
...consider the declaration Either<Empty, Either<Leaf, Node>>, which creates a three-part data structure...
Neal Ford on Immutability
(https://www.ibm.com/developerworks/java/library/j-ft4/index.html)
example class (no getters/setters/noArg constr; all flds final; class final):
public final class Address {
private final List streets;
public final String city;
public final String state;
public final String zip;
public Address(List streets, String city, String state, String zip) {
this.streets = streets;
this.city = city;
this.state = state;
this.zip = zip;
}
public final List getStreets() {
return Collections.unmodifiableList(streets);
}
}
Neal Ford's ex using a memoized cache
(https://www.ibm.com/developerworks/java/library/j-ft17/index.html)
"...Here, I create a class-wide cache to hold previously calculated values
....requires a little dubious class design, forcing it to be stateful
so that the instance can act as the owner of the cache. This could
be improved, but the improvement is trivial in subsequent examples,
so I won't bother here...."
import fj.F;
import fj.data.List;
import java.util.HashMap;
import java.util.Map;
import static fj.data.List.range;
import static fj.function.Integers.add;
import static java.lang.Math.round;
import static java.lang.Math.sqrt;
public class FjPrimeNumber {
private int candidate;
private Map cache;
public FjPrimeNumber setCandidate(int value) {
this.candidate = value;
return this;
}
public FjPrimeNumber(int candidate) {
this.candidate = candidate;
cache = new HashMap();
}
public boolean isFactor(int potential) {
return candidate % potential == 0;
}
public List getFactors() {
final List lowerFactors = range(1, (int) round(sqrt(candidate) + 1))
.filter(new F() {
public Boolean f(final Integer i) {
return isFactor(i);
}
});
return lowerFactors.append(lowerFactors.map(new F() {
public Integer f(final Integer i) {
return candidate / i;
}
}))
.nub();
}
public int sumFactors() {
if (cache.containsKey(candidate))
return cache.get(candidate);
else {
int sum = getFactors().foldLeft(add, 0);
cache.put(candidate, sum);
return sum;
}
}
public boolean isPrime() {
return candidate == 2 || sumFactors() == candidate + 1;
}
}
ibm devWorks: Neal Ford's Fluent interfaces
Evolutionary architecture and emergent design: Fluent interfaces
Build internal DSLs to capture idiomatic domain patterns
Neal Ford
July 13, 2010
This installment of Evolutionary architecture and emergent design continues the discussion ofharvesting techniques for idiomatic patterns in emergent design. When you identify a reusablepattern, you should capture it in a way to sets it apart from the rest of your code. Domain-specific languages (DSLs) offer many techniques for concisely capturing data and functionality.This month, Neal Ford shows you three ways to build internal DSLs that capture idiomaticdomain patterns.
The preceding installment of this series introduced the subject of using domain-specificlanguages (DSLs) to capture domain idiomatic patterns. This installment continues with that topic,demonstrating various DSL construction techniques.
In his upcoming book Domain Specific Languages, Martin Fowler differentiates between two typesof DSLs (see Related topics). External DSLs build a new language grammar, requiring tools likelexx and yacc or Antlr. An internal DSL builds new languages atop a base language, whose syntaxit borrows and stylizes. The examples in this installment build internal DSLs using Java as the base language, constructing new mini-languages on top of its syntax.
About this series
This series aims to provide a fresh perspective on the often-discussed but elusive conceptsof software architecture and design. Through concrete examples, Neal Ford gives you asolid grounding in the agile practices of evolutionary architecture and emergent design. Bydeferring important architectural and design decisions until the last responsible moment, youcan prevent unnecessary complexity from undermining your software projects.
Underlying all the following techniques for constructing DSLs is the notion of implicit context.DSLs (especially internal ones) try to eliminate noisy syntax by creating contextual wrappersaround related elements. A good example of this concept appears in XML in the form of parentand child elements, which provide a wrapper around related items. You'll notice many of theseDSL techniques achieve that same effect using language syntactic tricks.
Readability is one of the benefits of using a DSL. If you write code that nondevelopers can read,you shorten the feedback loop between your team and the people who are requesting features.A common DSL pattern identified in Fowler's book is called fluent interface, which he defines asbehavior capable of relaying or maintaining the instruction context for a series of method calls. I'llshow you several types of fluent interfaces, starting with method chaining.
Method chaining
Method chaining uses return values from methods to relay instruction context, which in this case isthe object instance making the first method invocation. This sounds much more complex than it is,so I'll show an example to clarify this concept.
When working with DSLs, it is common to start with your goal syntax and reverse engineerbackwards to figure out how to implement it. Starting at the end makes sense because readabilityis highly valued in DSLs. The example I'll use is a small application that tracks calendar entries.The application illustrates the DSL's syntax, as shown in Listing 1:
Listing 1. Goal syntax for a calendar DSL
public class CalendarDemoChained {
public static void main(String[] args) {
new CalendarDemoChained();
}
public CalendarDemoChained() {
Calendar fourPM = Calendar.getInstance();
fourPM.set(Calendar.HOUR_OF_DAY, 16);
Calendar fivePM = Calendar.getInstance();
fivePM.set(Calendar.HOUR_OF_DAY, 17);
AppointmentCalendarChained calendar = new AppointmentCalendarChained();
calendar.add("dentist").
from(fourPM).
to(fivePM).
at("123 main street");
calendar.add("birthday party").
at(fourPM);
displayAppointments(calendar);
}
private void displayAppointments(AppointmentCalendarChained calendar) {
for (Appointment a : calendar.getAppointments())
System.out.println(a.toString());
}
}
After the necessary cruft at the top dealing with Java calendars, you can see the method-chainingfluent interface in action as I add values to the two calendar entries. Notice that I'm using whitespace to separate the parts of what is (from a Java syntax standpoint) a single line of code. It iscommon in internal DSLs to stylize the use of the base language to make the DSL more readable.
The Appointment class containing most of the fluent-interface methods appears in Listing 2:
As you can see, building fluent interfaces is straightforward. For each of the mutator methods, youdiverge from standard JavaBean syntax by writing your setter methods to return the host object(this) and by replacing the set naming convention with something more readable. The generaldefinition at the start of this section should now be clear. The context being relayed via the methodchaining is this, meaning that you can make a series of method calls concisely.
Car2 car = new CarImpl();
MarketingDescription desc = new MarketingDescriptionImpl();
desc.setType("Box");
desc.setSubType("Insulated");
desc.setAttribute("length", "50.5");
desc.setAttribute("ladder", "yes");
desc.setAttribute("lining type", "cork");
car.setDescription(desc);
The problem domain for train cars is complex because of regulatory rules about contents andhistory. On the project that yielded this example, we had lots of complicated testing scenarios thatrequired dozens of lines of set calls like the ones in Listing 3. We tried to get our business analyststo verify that we had the right magical combination of attributes, but they pushed back becausethey viewed it as Java code, which they had no interest in reading. The ultimate impact of thisproblem was the requirement that a developer verbally translate the details, which is of courseerror-prone and time-consuming.
To resolve this problem, we converted our Car class into a fluent interface, so that the code fromListing 3 became the fluent interface shown in Listing 4:
Listing 4. Fluent interface for train cars
Car car = Car.describedAs()
.box()
.length(50.5)
.type(Type.INSULATED)
.includes(Equipment.LADDER)
.lining(Lining.CORK);
This code was declarative enough and removed sufficient noise from the Java API version that ourbusiness analysts were happy to verify it for us.
Returning to the calendar example, the last bit of the implementation is the AppointmentCalendarclass, which appears in Listing 5:
Listing 5. AppointmentCalendar
public class AppointmentCalendarChained {
private List appointments;
public AppointmentCalendarChained() {
appointments = new ArrayList();
}
public List getAppointments() {
return appointments;
}
public Appointment add(String name) {
Appointment appt = new Appointment(name);
appointments.add(appt);
return appt;
}
}
The add() method:
1. Starts the method chain by creating a new Appointment instance
2. Adds the new instance to the list of appointments
3. Ultimately returns the new appointment instance, meaning that subsequent method calls areinvoked on the new appointment
When you run the application, you see the details of your configured appointments, as shown inFigure 1:
Figure 1. Results of running the calendar application
So far, method chaining looks like a simple way to clean up overly verbose syntax, especiallymethod calls that are mostly declarative. This works well for idiomatic patterns in emergent designbecause domain patterns are frequently declarative.
Note that using method chaining necessitates violating the syntactic rules for JavaBeans, whichinsist that mutator methods must start with set and return void. Building fluent interfaces is anexample of knowing when it makes sense to break some of the rules. The JavaBeans specificationisn't doing you any favors if it forces you to write obfuscated code! But nothing in the creation oruse of fluent interfaces precludes supporting both the fluent interface and a JavaBeans interface.The fluent interface methods can turn around and call the standard set methods, allowing you touse fluent interfaces even when frameworks insist that it interact with your classes as JavaBeans.
Solving the finishing problem
One pitfall that's inherent in fluent interfaces under certain circumstances is known as the finishingproblem. I'll illustrate this issue by making a change to the AppointmentCalendar class fromListing 5. Presumably, you'd like to do more than just display the appointments, such as put themin a database or some other persistence mechanism. Where do you add the code to save thecompleted appointment to storage? You can try to do it in AppointmentCalendar's add() methodjust before you return the appointment. Listing 6 shows an attempt to access the appointment herefor something as simple as printing it:
Listing 6. Adding printing
public Appointment add(String name) {
Appointment appt = new Appointment(name);
appointments.add(appt);
System.out.println(appt.toString());
return appt;
}
When you run the code in Listing 6, the unhappy results illustrated in Figure 2 show up:
Figure 2. Error output after the addition to AppointmentCalendar
(nullpointerExc)
The error displayed is a NullPointerException occurring in the toString() method on theAppointment class. Why it complains here, even though the method worked correctly, is theessence of the finishing problem.
The error occurs because I'm trying to call the toString() method on the appointment instancebefore the rest of the fluent-interface setter methods are called. The code to try printing theappointment appears in the method that creates the appointment instance and starts the chain. Icould create a save() or finished() method that must be called as the last method in the chain,but I'd rather not impose an easy-to-forget rule on my DSL's users. In fact, I'd rather not imposeany order semantics on the methods in my fluent interface.
The real problem is that I'm being too aggressive with the method-chaining technique. Methodchaining works best for the creation of simple data objects, yet here I'm using it both for the settermethods on Appointment and in AppointmentCalendar to start the method chain.
I can fix the finishing problem by wrapping the creation of the appointment entirely with theparentheses of the appointment calendar's add() method, as shown Listing 7:
Listing 7. Wrapping via parameter
AppointmentCalendar calendar = new AppointmentCalendar();
calendar.add(
new Appointment("Dentist").
at(fourPM));
calendar.add(
new Appointment("Conference Call").
from(fourPM).
to(fivePM).
at("555-123-4321"));
calendar.add(
new Appointment("birthday party").
from(fourPM).
to(fivePM)).
add( new Appointment("Doctor").
at("123 Main St"));
calendar.add( new Appointment("No Fluff, Just Stuff").
at(fourPM));
displayAppointments(calendar);
In Listing 7, the add() method's parentheses encapsulate the entire use of the Appointmentfluent interface, allowing the add() method to handle whatever additional behavior that it wants(printing, persistence, and so on). In fact, I couldn't resist adding a bit of a fluent interface toAppointmentCalendar itself: you can now chain together the add() methods, as shown in Listing 7and implemented in Listing 8:
Listing 8. The parameter-wrapping AppointmentCalendar
public class AppointmentCalendar {
private List appointments;
public AppointmentCalendar() {
appointments = new ArrayList();
}
public AppointmentCalendar add(Appointment appt) {
appointments.add(appt);
return this;
}
public List getAppointments() {
return appointments;
}
}
The finishing problem can arise any time you mix fluent-interface classes. It popped up inthis example because I used the appointment calendar to start the method chain, mixingthe construction and wrapping behaviors. By deferring construction and initialization tothe Appointment class, I make it easier to separate additional wrapping behavior (such aspersistence).
Wrapping via functional sequence
Thus far, I've shown two of the three context-passing techniques for fluent-interface DSLs. The third functional sequence uses inheritance and anonymous inner classes to create a contextwrapper. The calendar application rewritten using functional sequence appears in Listing 9:
Listing 9. Wrapping via functional sequence
Listing 9 shows a pattern that I introduced in "Leveraging reusable code, Part 2" under the guiseof removing structural duplication. The syntax looks odd because of the double {{ braces. The firstset of enclosing braces delineates the construction of an anonymous inner class, and the secondset delineates the instance initializer for the anonymous inner class. (If this sounds a bit confusing,you can refer back to "Leveraging Reusable Code, Part 2" for a lengthy explanation of this Javaidiom.)
The main advantage of this style of fluent interface lies in its adaptability. The only thing a classneeds to be used this way is a default constructor (which allows you to create an anonymous innerclass instance inheriting from your class). That means that you can easily add fluent-interface
methods to existing Java APIs without changing any of the current calling semantics. This allowsyou to "fluentize" existing APIs gradually.
Conclusion
DSLs capture idiomatic domain patterns concisely and effectively. Fluent interfaces provide asimple way to change the way you write code so that you can more readily see the idiomaticpatterns you've fought to identify. They also force you to change your perspective on code just abit: it should be not merely functional but readable as well, especially if nondevelopers need toconsume any aspect of it. Fluent interfaces remove unnecessary noise from your syntax, allowingfor more readable code. For declarative structures, you can express ideas more clearly with lesseffort.
In the next installment, I'll continue discussing DSL techniques as a mechanism for harvestingidiomatic patterns in emergent design.
LocalTime leetTime = LocalTime.parse("13:37", germanFormatter); System.out.println(leetTime); // 13:37 /*** * Excerpted from "Functional Programming in Java", * published by The Pragmatic Bookshelf. * Copyrights apply to this code. It may not be used to create training material, * courses, books, articles, and the like. Contact us if you are in doubt. * We make no guarantees that this code is fit for any purpose. * Visit http://www.pragmaticprogrammer.com/titles/vsjava8 for more book information. ***/
@SuppressWarnings("unchecked") public class Camera { private Function<Color, Color> filter;
public Color capture(final Color inputColor) { final Color processedColor = filter.apply(inputColor); //... more processing of color... return processedColor; }
public static void main(final String[] args) { final Camera camera = new Camera(); final Consumer<String> printCaptured = (filterInfo) -> System.out.println(String.format("with %s: %s", filterInfo, camera.capture(new Color(200, 100, 200))));
} ----------------OUTPUT------------------- with no filter: java.awt.Color[r=200,g=100,b=200] with brighter filter: java.awt.Color[r=255,g=142,b=255] with darker filter: java.awt.Color[r=140,g=70,b=140] with brighter & darker filter: java.awt.Color[r=200,g=100,b=200] package fpij;
@SuppressWarnings("unchecked") public class Camera2 { private Function<Color, Color> filter;
public Color capture(final Color inputColor) { final Color processedColor = filter.apply(inputColor); //... more processing of color... return processedColor; }
} ----------------OUTPUT------------------- with no filter: java.awt.Color[r=200,g=100,b=200] with brighter filter: java.awt.Color[r=255,g=142,b=255] with darker filter: java.awt.Color[r=140,g=70,b=140] with brighter & darker filter: java.awt.Color[r=200,g=100,b=200] package fpij;
public class OlderThan20 { public static void main(String[] args) { final List<Person> people = Arrays.asList( new Person("John", 20), new Person("Sara", 21), new Person("Jane", 21), new Person("Greg", 35));
} ----------------OUTPUT------------------- getting length for Brad getting length for Kate getting length for Kim converting to uppercase: Kim KIM Stream created, filtered, mapped... ready to call findFirst... getting length for Brad getting length for Kate getting length for Kim converting to uppercase: Kim KIM package fpij;
public class Heavy { public Heavy() { System.out.println("Heavy created"); }
public String toString() { return "quite heavy"; } }
public class HolderNaive { private Heavy heavy;
public HolderNaive() { System.out.println("Holder created"); }
public Heavy getHeavy() { if(heavy == null) { heavy = new Heavy(); }
return heavy; }
//...
public static void main(final String[] args) { final HolderNaive holder = new HolderNaive(); System.out.println("deferring heavy creation..."); System.out.println(holder.getHeavy()); System.out.println(holder.getHeavy()); } } ----------------OUTPUT------------------- Holder created deferring heavy creation... Heavy created quite heavy quite heavy package fpij;
import java.util.function.Supplier;
public class Holder { private Supplier<Heavy> heavy = () -> createAndCacheHeavy();
public Holder() { System.out.println("Holder created"); }
public Heavy getHeavy() { return heavy.get(); } //...
private synchronized Heavy createAndCacheHeavy() { class HeavyFactory implements Supplier<Heavy> { private final Heavy heavyInstance = new Heavy();
public Heavy get() { return heavyInstance; } }
if(!HeavyFactory.class.isInstance(heavy)) { heavy = new HeavyFactory(); }
return heavy.get(); }
public static void main(final String[] args) { final Holder holder = new Holder(); System.out.println("deferring heavy creation..."); System.out.println(holder.getHeavy()); System.out.println(holder.getHeavy()); } }
----------------OUTPUT------------------- Holder created deferring heavy creation... Heavy created quite heavy quite heavy
package fpij;
import java.util.function.Supplier;
public class Evaluation { public static boolean evaluate(final int value) { System.out.println("evaluating ..." + value); simulateTimeConsumingOp(2000); return value > 100; } //...
public static void simulateTimeConsumingOp(final int millseconds) { try { Thread.sleep(2000); } catch(Exception ex) { throw new RuntimeException(ex); } }
public static void eagerEvaluator( final boolean input1, final boolean input2) { System.out.println("eagerEvaluator called..."); System.out.println("accept?: " + (input1 && input2)); }
public static void lazyEvaluator( final Supplier<Boolean> input1, final Supplier<Boolean> input2) { System.out.println("lazyEvaluator called..."); System.out.println("accept?: " + (input1.get() && input2.get())); }
public class RodCutterMemoized extends RodCutterBasic { public RodCutterMemoized(final List<Integer> pricesForLength) { super(pricesForLength); }
public int maxProfit(final int rodLength) { BiFunction<Function<Integer, Integer>, Integer, Integer> compute = (func, length) -> { int profit = (length <= prices.size()) ? prices.get(length - 1) : 0; for(int i = 1; i < length; i++) { int priceWhenCut = func.apply(i) + func.apply(length - i); if(profit < priceWhenCut) profit = priceWhenCut; } return profit; }; return callMemoized(compute, rodLength); }
public static void main(final String[] args) { final RodCutterMemoized rodCutterMomoized = new RodCutterMemoized(RodCutterBasic.priceValues); //run(foo);
System.out.println(rodCutterMomoized.maxProfit(5)); System.out.println(rodCutterMomoized.maxProfit(22)); } } public class Memoizer { public static <T, R> R callMemoized( final BiFunction<Function>T,R <, T, R< function, final T input) { Function<T, R> memoized = new Function<T, R>() { private final Map<T, R> store = new HashMap<>(); public R apply(final T input) { return store.computeIfAbsent(input, key -> function.apply(this, key)); } }; return memoized.apply(input); } } ----------------OUTPUT------------------- 10 44
import java.util.stream.Stream;
@FunctionalInterface public interface TailCall<T> {
public class Factorial { public static int factorialRec(final int number) { if(number == 1) return number; else return number * factorialRec(number - 1); }
public static TailCall<Integer> factorialTailRec( final int factorial, final int number) { if (number == 1) return done(factorial); else return call(() -> factorialTailRec(factorial * number, number - 1)); }
public static int factorial(final int number) { return factorialTailRec(1, number).invoke(); }
public class BigFactorial { public static BigInteger decrement(final BigInteger number) { return number.subtract(BigInteger.ONE); }
public static BigInteger multiply( final BigInteger first, final BigInteger second) { return first.multiply(second); }
final static BigInteger ONE = BigInteger.ONE; final static BigInteger FIVE = new BigInteger("5"); final static BigInteger TWENTYK = new BigInteger("20000");
def validateName(str: String): Validated[Name] =
cond(str.length <= 3, Name(str), List("your name is too lo.."))
def validateAge(i: Int): Validated[Age] =
cond(i >= 18, Age(i), List("bring your parents!"))
def validateTlf(str: String): Validated[Telephone] =
cond(str.length == 8, Telephone(str), List("No answer!"))
Because Either is a Monad in Scala we can implement our validation function using the map and flatMap properties provided by the Monad type class:
def validateUser(name: String, age: Int, tel: String): Validated[User] =
validateName(name).flatMap(
n =>
validateAge(age).flatMap(
a =>
validateTlf(tel)
.map(t => User(n, a, t))
)
)
Using for-comprehensions, which are just syntax sugar for flatMap and map we get very nice syntax for defining a sequence of monadic operations:
def validateUser(name: String, age: Int, tel: String): Validated[User] =
for {
n <- validateName(name)
a <- validateAge(age)
t <- validateTlf(tel)
} yield User(n, a, t)
Using the for-comp.'s in Java? SO consensus is 'use flatMap/map; this is just syntactical sugar'
https://medium.com/beingprofessional/think-functional-rethinking-criteria-pattern-with-lambdas-2bac1d77fe7d
...
we have written two separate predicates for two different values of gender. What if the set of possible values of a variable is higher ? We dont want to write a separate predicate for each and every value. This can be fixed by writing a function which takes a value and returns a predicate which compares the value with the actual value of the field. This is known as currying.
Function> genderFilter =
gender -> person -> person.getGender().equals(gender);
Then we can generate required predicate for a specific value (gender) according to our need.
https://medium.com/@johnmcclean/lambdas-are-not-functional-programming-63533ce2eb74
A Java long read : Is functional programming worth it?
https://medium.com/@johnmcclean/a-java-long-read-is-functional-programming-worth-it-ca53bfcd0c6a
"Better" is obviously subjective but Evan Farrer did
his master's
thesis on rewriting python programs into Haskell
(a purely functional language) and seeing how many
bugs it caught. You can read his paper, but the
Haskell compiler caught many non-trivial bugs...
Note comment:
...He found Haskell caught bugs because it had 'powerful'
type checking, not because it was a functional
language. This answer is misleading.
Functional References - optics
Debasish Ghosh, in in the book Functional And Reactive Domain Modeling, defined a compose function that allows us to put together Lenses and reuse code:
ConversionResult c = new ConversionResult<>(new Composite(), warning1);
ConversionResult p = new ConversionResult<>(new Piece(), warning2);
c.forBoth(p, (_c, _p) -> _c.piece = _p);
//the value contained in c has been composed with the value contained in p
//c now contai
ConversionResult c = new ConversionResult<>(new Composite(), warning1);
//Note that c.generatePiece return an object of type ConversionResult.
We will assume it returns with warning2.
ConversionResult a = new ConversionResult<>(new Param(), warning3);
We did, however, create a convenience function which we call mapBoth,
which simplifies a common but complex pattern:
//p2 is identical to p1, but doesn't need nested function calls
FOLD
Of interest to us is the API of Either.fold which I will illustrate
with a simple example. fold takes two parameters, a leftMapper
function to transform the left value, if the value is a left type,
and a rightMapper function to transform the right value, should it
be a right type. In the example below, we transform an
Either type to another Either
by using the fold method and lambda functions as parameters.
EitherdoSomething() {
// dummy method stub
}
Either foldResult = doSomething().fold((someString) -> {
// what to return when the result of doSomething() is a string (on the left)
return Either.left(someString.length());
}, (someInteger) -> {
// what to return when the result of doSomething() is a integer (on the right)
return Either.right(Integer.toString(someInteger));
});
from vavr API
io.vavr.control Interface Either
superInterfaces -> Iterable, Serializable, Value
Either represents a value of two possible types. An Either is either a Either.Left or a Either.Right.
If the given Either is a Right and projected to a Left, the Left operations have no effect on the Right value.
If the given Either is a Left and projected to a Right, the Right operations have no effect on the Left value.
If a Left is projected to a Left or a Right is projected to a Right, the operations have an effect.
Example: A compute() function, which results either in an Integer value
(in the case of success) or in an error message of type String (in the case of failure).
By convention the success case is Right and the failure is Left.
Either value = compute().right().map(i -> i * 2).toEither();
default R getOrElseGet(Function other)
Gets the Right value or an alternate value, if the projected Either is a Left.
default Either bimap(Function leftMapper,
Function rightMapper)
Maps either the left or the right side of this disjunction.
default R getOrElseGet(Function other)
Gets the Right value or an alternate value, if the projected Either is a Left.
default U fold(Function leftMapper,
Function rightMapper)
Folds either the left or the right side of this disjunction.
Type Parameters:
U - type of the folded value
Parameters:
leftMapper - maps the left value if this is a Left
rightMapper - maps the right value if this is a Right
Functional Programming in JavaScript, Part 3: Introduction to Functors and Monads
Using map & flatMap
Unfolding multiple Optional variables in Java 8
I have the following problem. Let's say you have 2 Optional variables
Optional c1 = ...
Optional c2 = ...
and a method which needs 2 variables of type Contact
void match(Contact c1, Contact c2) {...}
and you need to unwrap both c1 and c2 Optional vars and pass them into the match() method.
"Is this the most elegant way to do that in Java 8?"
if (c1.isPresent() && c2.isPresent()) {
match(c1.get(), c2.get());
}
A: Solution you provided in scala is just syntax sugar for using flatMaps internally. You can use flatmaps in Java 8 too (but there are no syntax sugar for it).
c1.flatMap(contact1 -> c2.flatMap(contact2 -> match(c1,c2)))
A: It has to be ifPresent since match has return type void, and if match will return f.e. boolean it could be Optional result = c1.flatMap(contact1 -> c2.map(contact2 -> match(contact1 ,contact2))) klappvisor Apr 6 '16 at 11:35
A: Actually, a better way to do it in scala (and without syntax sugar) is something like c1.zip(c2).map { case (a,b) => match(a,b) }. Does java have zip? Dima Apr 6 '16 at 13:56
A: If you consider the arguments not having values as an exception then you could handle them like:
try {
match(c1.orElseThrow(NoVal::new), c2.orElseThrow(NoVal::new));
} catch (NoVal ex) {
...
}
If they aren't an exceptional case then I would go with your first option as being more explicit about your intent. In my view it's pretty clear and readable, and easy to change to use orElse if you wish to switch to using defaults if the optionals are empty.
answered Apr 6 '16 at 11:53
sprinter
What's the difference between map and flatMap methods in Java 8?
A: The type signature kinda tells the whole story.
map :: Stream T -> (T -> R) -> Stream R,
flatMap :: Stream T -> (T -> Stream R) -> Stream R. Chris Martin Oct 31 '14 at 22:56
A: @michael That type signature looks like Haskell, not Java. But it's not clear whether the actual Java signature is any more readable:
Stream flatMap(Function> mapper). Stuart Marks Jul 20 '17 at 1:34
A: Both map and flatMap can be applied to a Stream and they both return a Stream. The difference is that the map operation produces one output value for each input value, whereas the flatMap operation produces an arbitrary number (zero or more) values for each input value.
This is reflected in the arguments to each operation.
The map operation takes a Function, which is called for each value in the input stream and produces one result value, which is sent to the output stream.
The flatMap operation takes a function that conceptually wants to consume one value and produce an arbitrary number of values. However, in Java, it's cumbersome for a method to return an arbitrary number of values, since methods can return only zero or one value. One could imagine an API where the mapper function for flatMap takes a value and returns an array or a List of values, which are then sent to the output. Given that this is the streams library, a particularly apt way to represent an arbitrary number of return values is for the mapper function itself to return a stream! The values from the stream returned by the mapper are drained from the stream and are passed to the output stream. The "clumps" of values returned by each call to the mapper function are not distinguished at all in the output stream, thus the output is said to have been "flattened."
Typical use is for the mapper function of flatMap to return Stream.empty() if it wants to send zero values, or something like Stream.of(a, b, c) if it wants to return several values. But of course any stream can be returned.
A: Try viewing it from a different perspective: it's not a transparent case where you can see the inner workings through, but the whole function itself is transparent, i.e. invisible, to you - while still doing their work and letting you see what you're working with. In this case, "flat" refers to the opposite of "nested", flatmap removes one nesting level by flattening. Zefiro
A: Flattening comes from turning 2-level structure into single level structure, see Dici's answer for an example stackoverflow.com/a/26684582/6012102 andrzej.szmukala Jan 16 at 11:42
A: Stream.flatMap, as it can be guessed by its name, is the combination of a map and a flat operation. That means that you first apply a function to your elements, and then flatten it. Stream.map only applies a function to the stream without flattening the stream.
To understand what flattening a stream consists in, consider a structure like [ [1,2,3],[4,5,6],[7,8,9] ] which has "two levels". Flattening this means transforming it in a "one level" structure : [ 1,2,3,4,5,6,7,8,9 ].
edited May 17 '16 at 21:43 asgs
A: I would like to give 2 examples to get a more practical point of view:
1st example making usage of map:
@Test
public void convertStringToUpperCaseStreams() {
List collected = Stream.of("a", "b", "hello") // Stream of String
.map(String::toUpperCase) // Returns a stream consisting of the results of applying the given function to the elements of this stream.
.collect(Collectors.toList());
assertEquals(asList("A", "B", "HELLO"), collected);
}
Nothing special in the first example, a Function is applied to return the String in uppercase.
Second example making usage of flatMap:
@Test
public void testflatMap() throws Exception {
List together = Stream.of(asList(1, 2), asList(3, 4)) // Stream of List
.flatMap(List::stream)
.map(integer -> integer + 1)
.collect(Collectors.toList());
assertEquals(asList(2, 3, 4, 5), together);
}
In the second example, a Stream of List is passed. It is NOT a Stream of Integer!
If a transformation Function has to be used (through map), then first the Stream has to be flattened to something else (a Stream of Integer).
If flatMap is removed then the following error is returned: The operator + is undefined for the argument type(s) List, int.
It is NOT possible to apply + 1 on a List of Integers!
edited Jan 12 at 22:42 Jan Nielsen
A: The function you pass to stream.map has to return one object. That means each object in the input stream results in exactly one object in the output stream.
The function you pass to stream.flatMap returns a stream for each object. That means the function can return any number of objects for each input object (including none). The resulting streams are then concatenated to one output stream.
answered Oct 31 '14 at 22:58 Philipp
A: for a Map we have a list of elements and a (function,action) f so :
[a,b,c] f(x) => [f(a),f(b),f(c)]
and for the flat map we have a list of elements list and we have a (function,action) f and we want the result to be flattened :
[[a,b],[c,d,e]] f(x) =>[f(a),f(b),f(c),f(d),f(e)]
answered Oct 8 '17 at 3:19 Bachiri Taoufiq Abderrahman
A: One line answer: flatMap helps to flatten a Collection> into a Collection. In the same way, it will also flatten an Optional> into Optional.
A: I have a feeling that most answers here overcomplicate the simple problem. If you already understand how the map works that should be fairly easy to grasp.
There are cases where we can end up with unwanted nested structures when using map(), the flatMap() method is designed to overcome this by avoiding wrapping.
Examples:
1
List> result = Stream.of(Arrays.asList(1), Arrays.asList(2, 3))
.collect(Collectors.toList());
We can avoid having nested lists by using flatMap:
List result = Stream.of(Arrays.asList(1), Arrays.asList(2, 3))
.flatMap(i -> i.stream())
.collect(Collectors.toList());
2
Optional> result = Optional.of(42)
.map(id -> findById(id));
Optional result = Optional.of(42)
.flatMap(id -> findById(id));
where:
private Optional findById(Integer id)
edited Apr 15 at 20:08
answered Jul 25 '17 at 17:41 Grzegorz Piwowarek
A: Oracle's article on Optional highlights this difference between map and flatmap:
String version = computer.map(Computer::getSoundcard)
.map(Soundcard::getUSB)
.map(USB::getVersion)
.orElse("UNKNOWN");
Unfortunately, this code doesn't compile. Why? The variable computer is of type Optional, so it is perfectly correct to call the map method. However, getSoundcard() returns an object of type Optional. This means the result of the map operation is an object of type Optional>. As a result, the call to getUSB() is invalid because the outermost Optional contains as its value another Optional, which of course doesn't support the getUSB() method.
With streams, the flatMap method takes a function as an argument, which returns another stream. This function is applied to each element of a stream, which would result in a stream of streams. However, flatMap has the effect of replacing each generated stream by the contents of that stream. In other words, all the separate streams that are generated by the function get amalgamated or "flattened" into one single stream. What we want here is something similar, but we want to "flatten" a two-level Optional into one.
Optional also supports a flatMap method. Its purpose is to apply the transformation function on the value of an Optional (just like the map operation does) and then flatten the resulting two-level Optional into a single one.
So, to make our code correct, we need to rewrite it as follows using flatMap:
String version = computer.flatMap(Computer::getSoundcard)
.flatMap(Soundcard::getUSB)
.map(USB::getVersion)
.orElse("UNKNOWN");
The first flatMap ensures that an Optional is returned instead of an Optional>, and the second flatMap achieves the same purpose to return an Optional. Note that the third call just needs to be a map() because getVersion() returns a String rather than an Optional object.
http://www.oracle.com/technetwork/articles/java/java8-optional-2175753.html
answered Feb 24 '17 at 19:54
Rusty Core
Using Flatmap/Filter
Q: I have a Stream of Integer and I would like to find the two numbers whose sum is equals to another number. So I came up with the following solution:
BiPredicate p = (price1, price2) -> price1.intValue() + price2.intValue() == moneyQty;
flavoursPrices.filter(p);
But the filter method does not receive a BiPredicate. Why not? What is an alternative for that?
A: You can still work with Bipredicate. The argument that the filter-method needs is a Predicate, so here is an example of how to use this BiPredicate:
BiPredicate p = (price1, price2) -> price1.intValue() + price2.intValue() == moneyQty;
flavoursPrices.stream().filter(el->p.test(el.price1,el.price2));
In this example flavoursPrices must be a List.
The lambda that we are using:
el->p.test(el.price1,el.price2)
Is replacing the anonymous inner class declaration for creating a new Predicate out of the BiPredicate:
Predicate arg =new Predicate() {
@Override
public boolean test(Element el) {
return p.test(el.price1,el.price2);
}
};
So to filter the stream, we are creating a new Predicate for every element coming from the stream and than we use this Predicate as argument to use it's test-method. The big advantage of this is, that we don't have to create enormous amounts of Predicates in advance, but we can pass every element in the lambda function and get it's attributes.
answered Dec 24 '16 at 9:46 Matt
*********** A: Because filter does not work that way, it only filters one collection at a time. For your task, you need something like Python's itertools.product; this can be implemented using flatMap. In Scala, this looks as short as:
prices.flatMap(p1 =>
prices.flatMap(p2 =>
if (p1 + p2 == 5) List((p1, p2)) else List()))
If you want to do it in (native) Java, something like this comes out:
import java.util.*;
import java.util.stream.*;
public class Main {
public static void main(String[] args) {
List prices = Arrays.asList(1,2,3,4,5,6,7,8,9);
List> result = prices.stream().flatMap(p1 ->
prices.stream().flatMap(p2 ->
p1 + p2 == 5 ? Stream.of(Arrays.asList(p1, p2)) : Stream.empty()
)
).collect(Collectors.toList());
System.out.println(result);
}
}
... which prints [[1, 4], [2, 3], [3, 2], [4, 1]]. But I wouldn't use that in reality ;) At least, a tuple class would be nice to have.
It is worth noting that flatMap one of the most powerful functions there is; almost all stream transformations (filter, map, cartesian product, flatten) can be implemented using it. And rest (like zip) is almost always a fold/reduce (or, rarely, an unfold).
answered Mar 15 '16 at 21:16 phg
Using Reduce
Q: Given a value foo and a Stream of Consumer void functions, what's the most concise way to apply each function to the value? Right now I have
consumers.forEach(c -> c.accept(foo));
which isn't terrible, but I suspect there might be some way to turn this inside out and do it with just method references (no explicit lambda). Possibly something with a singleton list and zip()?
A: You could reduce your consumers by means of the Consumer.andThen method and then apply the resulting consumer to the foo argument:
Stuff added Oct 13 ->
[[Note: much of Mar12 unread]]
site:medium.com "functional programming state" -react
Object-Oriented Programming The Trillion Dollar Disaster
Ilya Suzdalnitski
clojure: Map, Filter & Reduce Kris Jenkins
https://cstheory.stackexchange.com/questions/4007/how-do-i-choose-a-functional-dictionary-data-structure/4028?r=SearchResults&s=22|9.3553#4028
https://cstheory.stackexchange.com/questions/4126/how-are-imperative-languages-more-different-from-each-other-than-functional-lang/4128?r=SearchResults&s=35|7.6795#4128
Observable pattern applied in massive record update
suggests adding addBulk()/delBulk() to proc multiple recs & trigger only 1 notification.
Pierre-Yves Saumont's Functional-programming-in-java
src code-> https://www.manning.com/downloads/1124
(Ch4: Recursion, corecursion & memoization; 4.1.3: using tail call elimination...)
Stuff added Mar 12 ->
THIS ARTICLE BASICALLY ALLOWS LIST RECURSION W/O TailCalls USING FOLD
(to calc min/average values in a list)
John McClean impl.s Circular programming, a Haskell technique in Java
"...Java doesnt support Tail Recursion but, as iteration and recursion are
functionally equivalent we will attempt to implement circular programming
techniques in Java iteratively."
Using Fold to eval part lazily & in one pass instd of running algo twice
We can reference the minimum value lazily, as the first element in the data Tuple as follows (using v1 to access the first element of the Tuple2)
Eval minimum = Eval.later(()->data.v1);
(ie, 1st of Tuple var passed in)
The above defines a List that lazily refers to the result of an operation on that List.
The last step is to return the List of mimimums, to do this we can lazily map data, extracting the minimum result from the lazy Eval.
hackerNews java+functional (2018>)
Go presents itself as a high-performance language with simple concurency (or Java without the JVM)
...I don't need to know by how much Go lowers development costs because it makes no claims that it does, so I simply assume that it doesn't. That it is faster than Python, easy to learn in a day or two, and that it compiles down to a native executable -- are all trivial to verify. If you want, all of its claims are supported by plentiful data.
Generic Algoriths, or reducing any problem to a tree/graph algo
https://medium.com/@alonso.delarte/i-would-reword-it-to-no-one-is-doing-functional-programming-in-java-because-there-are-probably-a-e03188320131 (also gd other articles, incl: Understanding the generated hash code override in Java, line by line)
Functional Programming With Java: Immutability Better data structures with immutable state https://medium.com/better-programming/functional-programming-with-java-immutability-8dc748e85f9e
Think Functional: Rethinking criteria pattern with lambdas https://medium.com/@sujitkamthe/think-functional-rethinking-criteria-pattern-with-lambdas-2bac1d77fe7d
Stuff added Feb 20 ->Trees & tree-traversal
http://rosettacode.org/wiki/Tree_traversal#Java (Level.Order)
Processing rose-tree data structures
https://medium.com/@jxxcarlson/on-outlines-rose-trees-and-zippers-1decab0b139a **** (elm) ...a zipper is a structure endowed with a set of functions for building a tree, navigating it, and transforming it...
**** in the section 'TableOfContents' article mentions expanding/collapsing outline levels!!! ..."a rose tree with a subree that is in focus." ... uses elm's diff-merge strategy to handle updates
https://en.wikipedia.org/wiki/Rose_tree
http://erichgess.github.io/blog/2015/11/01/zippers-2-building-a-rose-tree-zipper/ http://erichgess.github.io/blog/2015/08/17/f-number-zippers/ (treeZipper @ bottom of pg)
(javaScr) https://github.com/brucou/functional-rose-tree (contains links to libs for manipulating tree structures) Uses lens for tree transversal etc. pruneWhen...
The body of the inner lambda uses both the parameter it receives (candidate)
and the parameter from the outer scope. That is, the body of the inner lambda
relies on both its parameter and its lexical scope or defining scope.
treat everything to the right of the first arrow as a black box:
a lambda expression returned by the outer lambda expression.
Type inference VERY IMP FOR "fluent" mapping (@bastaInterference)
(NOTE THIS IS A CAUSE OF VERY FREQUENT @FakeMonkeyBastaCompiler errs)
since the appMaker/codeStealer bastas can't use jdk8 they WANT explicit type declaration.
The next time they throw fakeA$$ errs, we can help em behave
the explicit type, now redundant, has quietly disappeared.
curry using cascadingLambdas
Function> add = x -> y -> x + y;
Function add5 = add.apply(5);
System.out.println(add5.apply(2)); //prints 7 and
compose(f, g) (x) = f(g(x))
public static Function compose(Function f, Function g) {
return x -> f.apply(g.apply(x));
}
public static void main(String[] args) {
Function sin_asin = compose(Math::sin, Math::asin);
System.out.println(sin_asin.apply(0.5)); // prints "0.5"
}
public class Tree{
private T value;
private Tree left;
private Tree right;
public void replaceAll(T value){
this.value = value;
if (left != null)
left.replaceAll(value);
if (right != null)
right.replaceAll(value);
}
}
Folds in javaScript; theory (incl folding for composition); list-of-lists to struct
map & fold in java: https://dzone.com/articles/thinking-functional
An anamorphism is a function that generates a sequence by repeated application of the function to its previous result.
These objects are used in functional programming as unfolds.
Motherload of links (to JAVA RosettaCode pgs)
http://rosettacode.org/wiki/Y_combinator http://rosettacode.org/wiki/Anonymous_recursion#Java
http://rosettacode.org/wiki/Ackermann_function#Java
Q. Is there a Java development workflow - either using IDE, command line build tool or anything of sorts - that allows for hot code redeploying multiple pieces of code without paying the heavy price of bundling?
A. There is a tool called DCEVM (which I've never used) that acts as a patch for the JVM to allow repeated class reloading DURING RUNTIME.
(used by https://github.com/HotswapProjects/HotswapAgent)
A. DCEVM supports enhanced class redefinitions and is available for current JDK7 and JDK8.
https://github.com/dcevm/dcevm/releases
http://ssw.jku.at/dcevm/
HotswapAgent is an free JRebel alternative and supports DCEVM in various Frameworks.
http://hotswapagent.org/
A. I believe your options are to patch the JVM with DCEVM or similar (if similar exists), use the jdb (or an abstraction of it such as what IDEs do) to reload classes from the command line debugger tool, or write up a secondary class loader with a hook that you as the developer/user can access during execution to let it know when to dump it's definitions and grab new ones. The last option giving you the most flexibility, but is not an lighthearted engineering task to do well.
A. Spring Loaded is a JVM AGENT for reloading class file changes whilst a JVM IS RUNNING. It transforms classes at loadtime to make them amenable to later reloading. Unlike 'hot code replace' which only allows simple changes once a JVM is running (e.g. changes to method bodies), Spring Loaded allows you to add/modify/delete methods/fields/constructors. The annotations on types/methods/fields/constructors can also be modified and it is possible to add/remove/change values in enum types.
Spring Loaded is usable on any bytecode that may run on a JVM, and is actually the reloading system used in Grails 2.
Once up and running what effectively happens is that any classes loaded from jar files (dependencies) are not treated as reloadable, whilst anything loaded from .class files on disk is made reloadable. Once loaded the .class file will be watched (once a second) and should a new version appear SpringLoaded will pick it up. Any live instances of that class will immediately see the new form of the object, the instances do not need to be discarded and recreated.
(mpt: nothing to do with "spring mvc")
https://github.com/spring-projects/spring-loaded
A. jdk11 possibly has this covered under JVMTI (a Hotspot update)
There's a new kid on the block, RelProxy
(https://github.com/jmarranz/relproxy/), it is open source, is not so advanced like JRebel but it can be used to freely change a subset of your code in runtime and reload it with almost no performance penalty and no need of context reloading (no session lost) in development and production if you want.answered Jan 30 '15
Post from last yr by someone from jRebel also mentions --
Fakereplace
is also an instrumenting agent, like JRebel but it has a lot less support for Java code changes (I assume) and the number of supported frameworks isn't as impressive.
Feenix
can do as much as the Java Instrumentation API allows it to do. Which basically means it doesn't really add value on top of standard HotSwap of the JVM. Same for AgentSmith
JavaWorld article "Add dynamic Java code to your application
"Write code that can respond to changes at runtime"
[[Basically monitors a dir; compiles; uses a Proxy for loading the classLoader & switching
to the new classLoader with the newly compiled/updated class. Transparent to the user, no
reboot. Only issue: STATE]]
JavaServer Pages (JSP) is a more flexible technology than servlets because it can respond to dynamic changes at runtime. Can you imagine a common Java class that has this dynamic capability too? It would be interesting if you could modify the implementation of a service without redeploying it and update your application on the fly.
The article explains how to write dynamic Java code. It discusses runtime source code compilation, class reloading, and the use of the Proxy design pattern to make modifications to a dynamic class transparent to its caller.
An example of dynamic Java code
Let's start with an example of dynamic Java code that illustrates what true dynamic code means and also provides some context for further discussions. Please find this example's complete source code in Resources.
The example is a simple Java application that depends on a service called Postman. The Postman service is described as a Java interface and contains only one method, deliverMessage():
public interface Postman {
void deliverMessage(String msg);
}
A simple implementation of this service prints messages to the console. The implementation class is the dynamic code. This class, PostmanImpl, is just a normal Java class, except it deploys with its source code instead of its compiled binary code:
public class PostmanImpl implements Postman {
private PrintStream output;
public PostmanImpl() {
output = System.out;
}
public void deliverMessage(String msg) {
output.println("[Postman] " + msg);
output.flush();
}
}
The application that uses the Postman service appears below. In the main() method, an infinite loop reads string messages from the command line and delivers them through the Postman service:
public class PostmanApp {
public static void main(String[] args) throws Exception {
BufferedReader sysin = new BufferedReader(new InputStreamReader(System.in));
// Obtain a Postman instance
Postman postman = getPostman();
while (true) {
System.out.print("Enter a message: ");
String msg = sysin.readLine();
postman.deliverMessage(msg);
}
}
private static Postman getPostman() {
// Omit for now, will come back later
}
}
Execute the application, enter some messages, and you will see outputs in the console such as the following (you can download the example and run it yourself):
[DynaCode] Init class sample.PostmanImpl
Enter a message: hello world
[Postman] hello world
Enter a message: what a nice day!
[Postman] what a nice day!
Enter a message:
Everything is straightforward except for the first line, which indicates that the class PostmanImpl is compiled and loaded.
Now we are ready to see something dynamic. Without stopping the application, let's modify PostmanImpl's source code. The new implementation delivers all the messages to a text file, instead of the console:
// MODIFIED VERSION
public class PostmanImpl implements Postman {
private PrintStream output;
// Start of modification
public PostmanImpl() throws IOException {
output = new PrintStream(new FileOutputStream("msg.txt"));
}
// End of modification
public void deliverMessage(String msg) {
output.println("[Postman] " + msg);
output.flush();
}
}
Shift back to the application and enter more messages. What will happen? Yes, the messages go to the text file now. Look at the console:
[DynaCode] Init class sample.PostmanImpl
Enter a message: hello world
[Postman] hello world
Enter a message: what a nice day!
[Postman] what a nice day!
Enter a message: I wanna go to the text file.
[DynaCode] Init class sample.PostmanImpl
Enter a message: me too!
Enter a message:
Notice [DynaCode] Init class sample.PostmanImpl appears again, indicating that the class PostmanImpl is recompiled and reloaded. If you check the text file msg.txt (under the working directory), you will see the following:
[Postman] I wanna go to the text file.
[Postman] me too!
Amazing, right? We are able to update the Postman service at runtime, and the change is completely transparent to the application. (Notice the application is using the same Postman instance to access both versions of the implementations.)
Four steps towards dynamic code
Let me reveal what's going on behind the scenes. Basically, there are four steps to make Java code dynamic:
Deploy selected source code and monitor file changes
Compile Java code at runtime
Load/reload Java class at runtime
Link the up-to-date class to its caller
Deploy selected source code and monitor file changes
To start writing some dynamic code, the first question we have to answer is, "Which part of the code should be dynamicthe whole application or just some of the classes?" Technically, there are few restrictions. You can load/reload any Java class at runtime. But in most cases, only part of the code needs this level of flexibility.
The Postman example demonstrates a typical pattern on selecting dynamic classes. No matter how a system is composed, in the end, there will be building blocks such as services, subsystems, and components. These building blocks are relatively independent, and they expose functionalities to each other via predefined interfaces. Behind an interface, it is the implementation that is free to change as long as it conforms to the contract defined by the interface. This is exactly the quality we need for dynamic classes. So simply put: Choose the implementation class to be the dynamic class.
For the rest of the article, we'll make the following assumptions about the chosen dynamic classes:
The chosen dynamic class implements some Java interface to expose functionality
The chosen dynamic class's implementation does not hold any stateful information about its client (similar to the stateless session bean), so the instances of the dynamic class can replace each other
Please note that these assumptions are not prerequisites. They exist just to make the realization of dynamic code a bit easier so we can focus more on the ideas and mechanisms.
With the selected dynamic classes in mind, deploying the source code is an easy task. Figure 1 shows the file structure of the Postman example.
PostmanExample
bin
dynacode
sample
PostmanImpl.java
src
We know "src" is source and "bin" is binary. One thing worth noting is the dynacode directory, which holds the source files of dynamic classes. Here in the example, there's only one filePostmanImpl.java. The bin and dynacode directories are required to run the application, while src is not necessary for deployment.
Detecting file changes can be achieved by comparing modification timestamps and file sizes. For our example, a check to PostmanImpl.java is performed every time a method is invoked on the Postman interface. Alternatively, you may spawn a daemon thread in the background to regularly check the file changes. That may result in better performance for large-scale applications.
Compile Java code at runtime
After a source code change is detected, we come to the compilation issue. By delegating the real job to an existing Java compiler, runtime compilation can be a piece of cake. Many Java compilers are available for use, but in this article, we use the Javac compiler included in Sun's Java Platform, Standard Edition (Java SE is Sun's new name for J2SE).
At the minimum, you can compile a Java file with just one statement, providing that the tools.jar, which contains the Javac compiler, is on the classpath (you can find the tools.jar under /lib/):
int errorCode = com.sun.tools.javac.Main.compile(new String[] {
"-classpath", "bin",
"-d", "/temp/dynacode_classes",
"dynacode/sample/PostmanImpl.java" });
The class com.sun.tools.javac.Main is the programming interface of the Javac compiler. It provides static methods to compile Java source files. Executing the above statement has the same effect as running javac from the command line with the same arguments. It compiles the source file dynacode/sample/PostmanImpl.java using the specified classpath bin and outputs its class file to the destination directory /temp/dynacode_classes. An integer returns as the error code. Zero means success; any other number indicates something has gone wrong.
The com.sun.tools.javac.Main class also provides another compile() method that accepts an additional PrintWriter parameter, as shown in the code below. Detailed error messages will be written to the PrintWriter if compilation fails.
// Defined in com.sun.tools.javac.Main
public static int compile(String[] args);
public static int compile(String[] args, PrintWriter out);
I assume most developers are familiar with the Javac compiler, so I'll stop here. For more information about how to use the compiler, please refer to Resources.
Load/reload Java class at runtime
The compiled class must be loaded before it takes effect. Java is flexible about class loading. It defines a comprehensive class-loading mechanism and provides several implementations of classloaders. (For more information on class loading, see Resources.)
The sample code below shows how to load and reload a class. The basic idea is to load the dynamic class using our own URLClassLoader. Whenever the source file is changed and recompiled, we discard the old class (for garbage collection later) and create a new URLClassLoader to load the class again.
// The dir contains the compiled classes.
File classesDir = new File("/temp/dynacode_classes/");
// Load class "sample.PostmanImpl" with our own classloader.
URLClassLoader loader1 = new URLClassLoader(
new URL[] { classesDir.toURL() }, parentLoader);
Class cls1 = loader1.loadClass("sample.PostmanImpl");
Postman postman1 = (Postman) cls1.newInstance();
/*
* Invoke on postman1 ...
* Then PostmanImpl.java is modified and recompiled.
*/
// Reload class "sample.PostmanImpl" with a new classloader.
URLClassLoader loader2 = new URLClassLoader(
new URL[] { classesDir.toURL() }, parentLoader);
Class cls2 = loader2.loadClass("sample.PostmanImpl");
Postman postman2 = (Postman) cls2.newInstance();
/*
* Work with postman2 from now on ...
* Don't worry about loader1, cls1, and postman1
* they will be garbage collected automatically.
*/
Pay attention to the parentLoader when creating your own classloader. Basically, the rule is that the parent classloader must provide all the dependencies the child classloader requires. So in the sample code, the dynamic class PostmanImpl depends on the interface Postman; that's why we use Postman's classloader as the parent classloader.
We are still one step away to completing the dynamic code. Recall the example introduced earlier. There, dynamic class reload is transparent to its caller. But in the above sample code, we still have to change the service instance from postman1 to postman2 when the code changes. The fourth and final step will remove the need for this manual change.
Link the up-to-date class to its caller
How do you access the up-to-date dynamic class with a static reference? Apparently, a direct (normal) reference to a dynamic class's object will not do the trick. We need something between the client and the dynamic classa proxy. (See the famous book Design Patterns for more on the Proxy pattern.)
Here, a proxy is a class functioning as a dynamic class's access interface. A client does not invoke the dynamic class directly; the proxy does instead. The proxy then forwards the invocations to the backend dynamic class. Figure 2 shows the collaboration.
When the dynamic class reloads, we just need to update the link between the proxy and the dynamic class, and the client continues to use the same proxy instance to access the reloaded class. Figure 3 shows the collaboration.
In this way, changes to the dynamic class become transparent to its caller.
The Java reflection API includes a handy utility for creating proxies. The class java.lang.reflect.Proxy provides static methods that let you create proxy instances for any Java interface.
The sample code below creates a proxy for the interface Postman. (If you aren't familiar with java.lang.reflect.Proxy, please take a look at the Javadoc before continuing.)
InvocationHandler handler = new DynaCodeInvocationHandler(...);
Postman proxy = (Postman) Proxy.newProxyInstance(
Postman.class.getClassLoader(),
new Class[] { Postman.class },
handler);
The returned proxy is an object of an anonymous class that shares the same classloader with the Postman interface (the newProxyInstance() method's first parameter) and implements the Postman interface (the second parameter). A method invocation on the proxy instance is dispatched to the handler's invoke() method (the third parameter). And handler's implementation may look like the following:
public class DynaCodeInvocationHandler implements InvocationHandler {
// Get an instance of the up-to-date dynamic class
Object dynacode = getUpToDateInstance();
// Forward the invocation
return method.invoke(dynacode, args);
}
}
The invoke() method obtains an instance of the up-to-date dynamic class and forwards the invocation to it. This may involve source code compilation and class reloading if the dynamic class's source file has been modified.
Now we have completed the Postman service's dynamic code. A client creates a proxy of the service and calls the method deliverMessage() through the proxy. Each invocation on the proxy is dispatched to the DynaCodeInvocationHandler class's invoke() method. In that method, the latest service implementation is first obtained, which may involve necessary source compilation and class reloading. Then, the invocation forwards to the implementation for real processing.
Put it all together
We have gone through all the tricks required for the dynamic Java code. It's time to put them together to build something reusable.
We can build a utility to ease the adoption of dynamic code by encapsulating the four steps mentioned above. The Postman example relies on such a utility called DynaCode. Remember the PostmanApp application and its omitted getPostman() method? It's time to show it:
public class PostmanApp {
public static void main(String[] args) throws Exception {
See how the getPostman() method creates a dynamic Postman service, creates an instance of DynaCode, specifies a source directory, and returns a proxy of some dynamic implementation. To use your own dynamic Java code, you simply need to write three lines of code. Everything else is taken care of inside DynaCode. Try it yourself (the DynaCode's source code is included in the example).
I won't go into further details of DynaCode, but as a review of the technologies we've talked about, take a look at the sequence diagram in Figure 4 to see how DynaCode works at a high level.
Conclusion
In this article, I introduced the idea of dynamic Java code and the steps required to realize it. I covered topics such as runtime source compilation, class reloading, and the Proxy design pattern. Though none of these topics are new, by putting them together, we create an interesting dynamic feature for common Java classes so they can be modified and refreshed at runtime just like JSP pages.
An example is provided for demonstration purposes in Resources. It includes a reusable utility called DynaCode that makes writing your own dynamic code much easier.
I'd like to conclude by discussing the values and applications of dynamic code: Dynamic code can respond quickly to on-demand change requests. It can be used to implement truly dynamic services and business rules that change from time to time, and to replace embedded scripts used by task nodes of workflows. Dynamic code also eases application maintenance and greatly reduces outages caused by application redeployments.
Li Yang joined IBM in April 2004 and is a Certified IBM Rational Consultant on Rational Unified Process, requirements management, and object-oriented analysis and design. He has experience with Java server-side technologies, Web architectures, Java EE projects, and Java SE system library development.
Resources
Download the Postman example (tested with Sun J2SE 1.4.2)
/*
* Copyright (c) 2013, 2017, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License version 2 only, as
* published by the Free Software Foundation.
*
* This code is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
* version 2 for more details (a copy is included in the LICENSE file that
* accompanied this code).
*
* You should have received a copy of the GNU General Public License version
* 2 along with this work; if not, write to the Free Software Foundation,
* Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
*
* Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
* or visit www.oracle.com if you need additional information or have any
* questions.
*/
package jdk.test.lib.compiler;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.URI;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import javax.tools.ForwardingJavaFileManager;
import javax.tools.FileObject;
import javax.tools.JavaCompiler;
import javax.tools.JavaCompiler.CompilationTask;
import javax.tools.JavaFileObject;
import javax.tools.JavaFileObject.Kind;
import javax.tools.SimpleJavaFileObject;
import javax.tools.StandardLocation;
import javax.tools.ToolProvider;
/**
* {@code InMemoryJavaCompiler} can be used for compiling a {@link
* CharSequence} to a {@code byte[]}.
*
* The compiler will not use the file system at all, instead using a {@link
* ByteArrayOutputStream} for storing the byte code. For the source code, any
* kind of {@link CharSequence} can be used, e.g. {@link String}, {@link
* StringBuffer} or {@link StringBuilder}.
*
* The {@code InMemoryCompiler} can easily be used together with a {@code
* ByteClassLoader} to easily compile and load source code in a {@link String}:
*
*
* {@code
* import jdk.test.lib.compiler.InMemoryJavaCompiler;
* import jdk.test.lib.ByteClassLoader;
*
* class Example {
* public static void main(String[] args) {
* String className = "Foo";
* String sourceCode = "public class " + className + " {" +
* " public void bar() {" +
* " System.out.println("Hello from bar!");" +
* " }" +
* "}";
* byte[] byteCode = InMemoryJavaCompiler.compile(className, sourceCode);
* Class fooClass = ByteClassLoader.load(className, byteCode);
* }
* }
* }
*
*/
public class InMemoryJavaCompiler {
private static class MemoryJavaFileObject extends SimpleJavaFileObject {
private final String className;
private final CharSequence sourceCode;
private final ByteArrayOutputStream byteCode;
public MemoryJavaFileObject(String className, CharSequence sourceCode) {
super(URI.create("string:///" + className.replace('.','/') + Kind.SOURCE.extension), Kind.SOURCE);
this.className = className;
this.sourceCode = sourceCode;
this.byteCode = new ByteArrayOutputStream();
}
@Override
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
return sourceCode;
}
@Override
public OutputStream openOutputStream() throws IOException {
return byteCode;
}
public byte[] getByteCode() {
return byteCode.toByteArray();
}
public String getClassName() {
return className;
}
}
private static class FileManagerWrapper extends ForwardingJavaFileManager {
private static final Location PATCH_LOCATION = new Location() {
@Override
public String getName() {
return "patch module location";
}
@Override
public boolean isOutputLocation() {
return false;
}
};
private final MemoryJavaFileObject file;
private final String moduleOverride;
public FileManagerWrapper(MemoryJavaFileObject file, String moduleOverride) {
super(getCompiler().getStandardFileManager(null, null, null));
this.file = file;
this.moduleOverride = moduleOverride;
}
@Override
public JavaFileObject getJavaFileForOutput(Location location, String className,
Kind kind, FileObject sibling)
throws IOException {
if (!file.getClassName().equals(className)) {
throw new IOException("Expected class with name " + file.getClassName() +
", but got " + className);
}
return file;
}
@Override
public Location getLocationForModule(Location location, JavaFileObject fo) throws IOException {
if (fo == file && moduleOverride != null) {
return PATCH_LOCATION;
}
return super.getLocationForModule(location, fo);
}
@Override
public String inferModuleName(Location location) throws IOException {
if (location == PATCH_LOCATION) {
return moduleOverride;
}
return super.inferModuleName(location);
}
@Override
public boolean hasLocation(Location location) {
return super.hasLocation(location) || location == StandardLocation.PATCH_MODULE_PATH;
}
}
/**
* Compiles the class with the given name and source code.
*
* @param className The name of the class
* @param sourceCode The source code for the class with name {@code className}
* @param options additional command line options
* @throws RuntimeException if the compilation did not succeed
* @return The resulting byte code from the compilation
*/
public static byte[] compile(String className, CharSequence sourceCode, String... options) {
MemoryJavaFileObject file = new MemoryJavaFileObject(className, sourceCode);
CompilationTask task = getCompilationTask(file, options);
if(!task.call()) {
throw new RuntimeException("Could not compile " + className + " with source code " + sourceCode);
}
return file.getByteCode();
}
private static JavaCompiler getCompiler() {
return ToolProvider.getSystemJavaCompiler();
}
private static CompilationTask getCompilationTask(MemoryJavaFileObject file, String... options) {
List opts = new ArrayList<>();
String moduleOverride = null;
for (String opt : options) {
if (opt.startsWith("--patch-module=")) {
moduleOverride = opt.substring("--patch-module=".length());
} else {
opts.add(opt);
}
}
return getCompiler().getTask(null, new FileManagerWrapper(file, moduleOverride), null, opts, null, Arrays.asList(file));
}
}
Feb12 2021: CLI-only Document level aspect hooks
Create queryCreate; postCreate
Open queryOpen; postOpen
Edit queryModeChng/postModeChng
Save querySave; postSave
Refresh? Delete queryDelete; postDelete
DEV aspect hooks for designEls:
WITHIN docAspect; dispatch on (desEl.docType) {
//trigger approp. behavior }
Db level (?? poss not necc)
Fetch? Update? or copy all docLvl aspects? + qryConnect/etc.?
custom Fields
Basic Field Data-Types
Text < 250 char Text /unlimited
Boolean Date TimeStamp
BinaryData (incl opt for user to attach any file)
Number Int Number Long Number - Currency
Calculated (supply Formulas for user to use) (Formula based on existing lookupFld value)
Lookup (supplied list of vals) -> user can choose UI widgets (chckbox/radio/etc) Lookup (Tbl Column vals - restricted - only admin users can edit list) (Tbl Column vals - unrestricted - user can add to vals)
BSON is a binary serialization format used to store documents and make remote procedure calls in MongoDB. The BSON specification is located at bsonspec.org.
Each BSON type has both integer and string identifiers as listed in the following table:
Type Number Alias Notes Double 1 double String 2 string Object 3 object Array 4 array Binary data 5 binData Undefined 6 undefined Deprecated. ObjectId 7 objectId Boolean 8 bool Date 9 date Null 10 null Regular Expression 11 regex DBPointer 12 dbPointer Deprecated. JavaScript 13 javascript Symbol 14 symbol Deprecated. JavaScript (with scope) 15 javascriptWithScope 32-bit integer 16 int Timestamp 17 timestamp 64-bit integer 18 long Decimal128 19 decimal New in version 3.4. Min key -1 minKey Max key 127 maxKey You can use these values with the $type operator to query documents by their BSON type. The $type aggregation operator returns the type of an operator expression using one of the listed BSON type strings.
StackOverflow Q : Custom Fields for a Form representing an object
A: I want to answer your question in two parts: 1) Implementing custom fields in a database server 2) Restricting custom fields to an enumeration of values
Although common solutions to 1) are discussed in the question referenced by @Simon, maybe you are looking for a bit of discussion on what the problem is and why it hasn't been solved for us already.
databases are great for structured, typed data custom fields are inherently less structured therefore, custom fields are more difficult to work with in a database some or many of the advantages of using a database are lost some queries may be more difficult or impossible type safety may be lost (in the database) data integrity may no longer be enforced (by the database) it's a lot more work for the implementers and maintainers As discussed in the other question, there's no perfect solution. But these benefits/features still need to be implemented somewhere, and so often the application becomes responsible for data integrity and type safety. For situations like these, people have created Object-Relation Mapping tools, although, as Jeff Atwood says, even using an ORM could create more problems than it solved. However, you mentioned that it 'should be generic and be able to deal with all sorts of problems in the future' -- this makes me think an ORM might be your best bet.
So, to sum up my answer, this is a known problem with known solutions, none of which are completely satisfactory (because it's so hard). Pick your poison.
To answer the second part of (what I think is) your question: As mentioned in the linked question, you could implement Entity-Attribute-Value in your database for custom fields, and then add an extra table to hold the legal values for each entity. Then, the attribute/value of the EAV table is a foreign key into the attribute-value table.
For example,
CREATE TABLE `attribute_value` ( -- enumerations go in this table `attribute` varchar(30), `value` varchar(30), PRIMARY KEY (`attribute`, `value`) );
CREATE TABLE `eav` ( -- now the values of attributes are restricted `entityid` int, `attribute` varchar(30), `value` varchar(30), PRIMARY KEY (`entityid`, `attribute`), FOREIGN KEY (`attribute`, `value`) REFERENCES `attribute_value`(`attribute`, `value`) ); Of course, this solution isn't perfect or complete -- it's only supposed to illustrate the idea. For instance, it uses varchars, and lacks a type column. Also, who gets to decide what the possible values for each attribute are? Can these be changed at any time by the user?
shareimprove this answer edited May 23 '17 at 9:59
Community 11 answered Sep 23 '11 at 16:40
Matt Fenwick 29.9k12101165 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - When you let the users create the data models, I would recommend looking at an document database or 'NoSQL' since you want exactly that, to store schemaless data structures.
Also, sharePoint stores metadata the way you mentioned (10 columns for text, 5 for dates etc)
That said, in my current project (locked in SharePoint, so Framework 3.5 + SQL Server and all the constraints that follow) we use a somewhat similar structure as below:
Form Id
Attribute (or Field) Name Type (enum) Text, List, Dates, Formulas etc Hidden (bool) Mandatory DefaultValue Options (for lists) Readonly Mask (for SSN etc) Length (for text fields) Order
Metadata FormId AttributeId Text (the value for everything but dates) Date (the value for dates) Our formulas employ functions such as Increment: INC([attribute1][attribute2], 6) and this would produce something like 000999 for the 999th instance of the combined values for attribute 1 and attribute 2 for a form, this is stored as:
AttributeIncrementFormula AtributeId Counter Token Other 'formulas' (aka anything non-trivial) such as barcodes are stored as single metadata values. In the actual implementation, we would have something like this:
var form = formRepository.GetById(1); form.Metadata["firstname"].Value Value above is a readonly property that decides whether we should get the value from Text or Date and if some additional transform is required. Note that the database here is merely a storage, we hold all the domain complexity in the application.
We also let our customer decide which attribute is the form title for example, so if firstname is the form title, they'll set an in-memory param that spans the entire application to be something like Params.InMemory.TitleAttributeId = .
I hope this gives you some insight on a production impl of a similar scenario.
shareimprove this answer answered Sep 23 '11 at 17:16
Francisco Aquino 8,0712036 add a comment up vote 2 down vote
This is really more of a comment than an answer, but I need more space than SO will allow for comments, so here 'tis:
I think your UserConfiguration table approach is good, and would suggest only abstracting the "type" and "value" pieces of your design a bit more:
Since your application will need to validate user input, each notion of "type" will have an associated piece of evaluation logic. Obviously the more of this you can abstract into data the easier it will be to keep your code small. Enumerated lists are a good start, but if your "validator" logic can be extended to handle pattern matching for text strings and Boolean logical expressions (e.g. to describe/enforce constraints on input values), then you can express pretty much any "type" of input that your application may need to handle in terms of (relatively) simple "atoms" that you can map naturally to DB tables. When storing a user-specified value, you can either store the "raw" data (e.g. in JSON) and a foreign key to the associated "type", or you can add an lookup/cache system that assigns an integer to each new value that is encountered by the system ("novelty" can be checked by checking a hash of the "raw" data, for example). The latter approach obviously scales better if you're expecting lots of data duplication (which of course you would in the case of a multiple-choice menu). shareimprove this answer answered Sep 26 '11 at 18:11
Peter 2,0891827
Q: What are design patterns to support custom fields in an application?
We develop a commercial application. Our customers are asking for custom fields support. For instance, they want to add a field to the Customer form.
What are the known design patterns to store the field values and the meta-data about the fields?
I see these options for now:
Option 1: Add Field1, Field2, Field3, Field4 columns of type varchar to my Customer table.
Option 2: Add a single column of type XML in the customer table and store the custom fields' values in xml.
Option 3: Add a CustomerCustomFieldValue table with a column of type varchar and store values in that column. That table would also have a CustomerID, a CustomFieldID.
CustomerID, CustomFieldID, Value 10001, 1001, '02/12/2009 8:00 AM' 10001, 1002, '18.26' 10002, 1001, '01/12/2009 8:00 AM' 10002, 1002, '50.26' CustomFieldID would be an ID from another table called CustomField with these columns: CustomFieldID, FieldName, FieldValueTypeID.
Option 4: Add a CustomerCustomFieldValue table with a column of each possible value type and store values in the right column. Similar to #3 but field values are stored using a strongly-type column.
CustomerID, CustomFieldID, DateValue, StringValue, NumericValue 10001, 1001, 02/12/2009 8:00 AM, null, null 10001, 1002, null, null, 18.26 10002, 1001, 01/12/2009 8:00 AM, null, null 10002, 1002, null, null, 50.26 Option 5: Options 3 and 4 use a table specific to a single concept (Customer). Our clients are asking for custom field in other forms as well. Should we instead have a system-wide custom field storage system? So instead of having multiple tables such as CustomerCustomFieldValue, EmployeeCustomFieldValue, InvoiceCustomFieldValue, we would have a single table named CustomFieldValue? Although it seems more elegant to me, wouldn't that cause a performance bottleneck?
Have you used any of those approaches? Were you successful? What approach would you select? Do you know any other approach that I should consider?
Also, my clients want the custom field to be able to refer to data in other tables. For instance a client might want to add a "Favorite Payment Method" field to the Customer. Payment methods are defined elsewhere in the system. That brings the subject of "foreign keys" in the picture. Should I try to create constraints to ensure that values stored in the custom field tables are valid values?
Thanks
EDIT 07-27-2009:
Thank you for your answers. It seems like the list of approaches is now quite comprehensive. I have selected the option 2 (a single XML column). It was the easiest to implement for now. I will probably have to refractor to a more strongly-defined approach as my requirements will get more complex and as the number of custom fields to support will get larger.
database-design data-modeling shareimprove this question edited Jul 27 '09 at 21:41 asked Jul 14 '09 at 17:18
Sylvain 10.4k2282123
The simplest option (an XML column in each table) turned out good. 7 years later we are about to change the design. We are going with run time schema modification. When a client adds a field, we create a new column in the database on-the-fly, with the right data type and the proper foreign keys to other tables. The main reasons for this change: types (data integrity), foreign keys (data integrity), performance (indexes), clarity (querying and migration scripts are easier to write). XML was easy to implement and served us well; it is a viable option.
- Sylvain Mar 14 '16 at 19:13
I do agree with posters below that Options 3, 4, or 5 are most likely to be appropriate. However, each of your suggested implementations has its benefits and costs. I'd suggest choosing one by matching it to your specific requirements. For example:
Option 1 pros: Fast to implement. Allows DB actions on custom fields (searching, sorting.) Option 1 cons: Custom fields are generic, so no strongly-typed fields. Database table is inefficient, size-wise with many extraneous fields that will never be used. Number of custom fields allowed needs to be anticipated.
Option 2 pros: Fast to implement. Flexible, allowing arbitrary number and type of custom fields. Option 2 cons: No DB actions possible on custom fields. This is best if all you need to do is display the custom fields, later, or do minor manipulations of the data only on a per-Customer basis.
Option 3 pros: Both flexible and efficient. DB actions can be performed, but the data is normalized somewhat to reduce wasted space. I agree with unknown (google)'s suggestion that you add an additional column that can be used to specify type or source information. Option 3 cons: Slight increase in development time and complexity of your queries, but there really aren't too many cons, here.
Option 4 is the same as Option 3, except that your typed data can be operated on at the DB level. The addition of type information to the link table in Option 3 allows you to do more operations at our application level, but the DB won't be able to do comparisons or sort, for example. The choice between 3 and 4 depends on this requirement.
Option 5 is the same as 3 or 4, but with even more flexibility to apply the solution to many different tables. The cost in this case will be that the size of this table will grow much larger. If you are doing many expensive join operations to get to your custom fields, this solution may not scale well.
P.S. As noted below, the term "design pattern" usually refers to object-oriented programming. You're looking for a solution to a database design problem, which means that most advice regarding design patterns won't be applicable.
shareimprove this answer edited Jun 6 '12 at 16:11
Diederik 2,27823044 answered Jul 14 '09 at 18:18
Eric Nguyen 33k32030
I accept this answer because I think it helps choosing a solution.Sylvain Aug 13 '09 at 18:20
Regarding Option 3, "(...) but there really aren't too many cons, here." Not trying to remove any credibility from your answer but #3/4/5 are pretty much the Entity-Value-Attribute pattern, considered by some as an anti-pattern and there are plenty of "horror stories" about its performance around the Web. On the other hand, many say it's possible to search/sort/filter (@ server side) using XML/BLOB, but I have never seen it done... I'm scratching my head about which approach (EAV or XML) works best, especially when using an ORM.user1987392 Jul 4 '15 at 23:30
As far as the application code is concerned I'm unsure. I do know that custom fields benefit greatly from a EAV model in the database.
Per the comments below, the most significant mistake you can make with this model is putting foreign keys into it. Never ever put something like FriendID or TypeID into this model. Use this model in conjunction with the typical relational model and keep foreign key fields in table columns as they should.
A second significant mistake is placing data in this model that needs to be reported with every element. For example putting something like Username in this model would mean that anytime you want to access a user and need to know their username you've committed yourself to a join at best or 2n queries where n is the number of users you're looking at. When you consider that you are usually going to need the Username property for every User element it becomes obvious this too should remain in the table columns.
However, if you're just using this model with custom user fields you'll be fine. I can't imagine many situations where a user would be entering in relational data and the EAV model is not too significantly detrimental to searches.
Lastly, don't try to join data from this and get a nice pretty recordset. Grab the original record and then grab the set of records for the entity. If you find yourself tempted to join the tables you've probably made the second mistake as mentioned above.
shareimprove this answer edited Jul 14 '09 at 20:12 answered Jul 14 '09 at 17:23
Spencer Ruport 30.5k973124
I'm using this model in a current project of mine and it has been really beneficial. You also generate de-normalized views of the data when necessary for simple querying/data binding. - Dillie-O Jul 14 '09 at 17:26
Yup, when carefully applied these are pretty powerful. - Spencer Ruport Jul 14 '09 at 17:31
Pay close attention to the warnings in the Wikipedia article. Done poorly, entiry/value pairs can murder a system, as is attested by many disaster stories scattered about the web. If user-defined data has to relate to more items than just the "direct parent", you are probably much better off adding that data to the model as new rows or tables. My main piece of advice would be to try very hard not to overused such a system. - Philip Kelley Jul 14 '09 at 18:00
Interesting read on Wikipedia. EAV model is certainly a double edged sword. My +1 is putting a name to this approach and the fact that the wikipedia article is quite good. Personally I can see lots of pitfuls with following this approach, but can see where it would be useful. I think having an XML column is a serious alternative though, despite what the wikipedia article states. - RichardOD Jul 14 '09 at 19:53
I think everyone has made good points around this topic. For me: XML is just another representation of rows in a database. So adding an XML column is like having a tiny database inside a row. :(Diederik Jun 6 '12 at 13:30
If you're developing with an object oriented language, we're talking about adaptive object models here. There are quite a few articles written about how you can implement them in oo-languages, but not so much information about how to design the data store side.
In the company where I work, we have solved the problem by using a relational database to store AOM data. We have central entity table for presenting all the different "entities" in the domain, like people, network devices, companies, etc... We store the actual "form fields" to data tables that are typed, so we have one table for strings, one for dates and so on. All the data tables have a foreign key pointing to the entity table. We also need tables to present the type-side, i.e. what kind of attributes (form fields) can certain entity have and this information is used to interpret the data in data tables.
Pros of our solution are that anything can be modeled without code changes, including references between entities, multivalues and so on. It's also possible to add business rules and validations to fields and they can be reused in all form. Cons are that the programming model is not very easy to understand and query performance will be worse than with a more typical DB design. Some other solution than relational database could have been better and easier for AOM.
Building a good AOM with a working data store for it is a lot of work and I wouldn't recommend it if you don't have highly skilled developers. Maybe one day there will be an OS solution for these kinds of requirements.
Custom fields have been discussed before in SO:
* How would you create and store user-defined custom fields in a SQL database? (https://stackoverflow.com/questions/961232/how-would-you-create-and-store-user-defined-custom-fields-in-a-sql-database) * What architecture can I use to handle a shopping cart where each product requries different attributes to be saved? (https://stackoverflow.com/questions/287093/what-architecture-can-i-use-to-handle-a-shopping-cart-where-each-product-requries)
edited May 23 '17 at 12:34 Community answered Jul 14 '09 at 17:40 Kaitsu 3,07932235
Something like Option 3 is the way to go and i have used this method previously. Create a single table to define additional properties and their corresponding values. This would be a 1-N relationship between your Customer and CustomerCustomField table (respectively). Your second question regarding defining relationships with custom properties would be something to think about. The first thing that comes to mind is adding a DataSource field, which would contain the table to which the property value is bound to. So essentially your CustomerCustomField would look like:
CustomerId Property Value ValueDataSource (nullable) This should allow you to either bind to a specific data structure or simply allow you to specify unbound values. You can further normalize this model, but something like this could work and should be easy enough to handle in code.
shareimprove this answer answered Jul 14 '09 at 17:41 Sergey
Option 4 or 5 would be my choice. If your data is important, I wouldn't go tossing away your type information with Option 3. (You might try to implement full type-checking yourself, but it's a pretty big job, and the database engine already does it for you.)
Some thoughts:
*Make sure your CustomFields has a DataType column. - Use a UDF-based check constraint on CustomFieldValues to ensure that the column specified by CustomFields.DataType is non-null. - You'll also want a standard check constraint to make sure you have exactly one non-null value. *Regarding foreign keys, I would model these as a separate DataType. - Each potential cross-table reference would require its own column. This is good, because it maintains referential integrity. - You would have to support these relationships in application code anyway, so the fact that they are hard-coded in the database does not actually limit functionality. - This will also jive well with your ORM, if you're using one. *For Option 5, use intermediary tables to model the relationships. -You would still have a CustomerCustomFieldValue, but instead with only CustomerID and CustomFieldValueID columns. *Think long and hard about your constraints every step of the way. This is tricky stuff, and one misstep can cause utter havok down the line.
I am using this in an application currently in development. There haven't been any problems yet, but EAV designs still scare the daylights out of me. Just be careful.
As an aside, XML may also be a good choice. I don't know as much about it from direct experience, but it was one of the options I considered when starting the data design, and it looked pretty promising.
shareimprove this answer answered Jul 14 '09 at 18:00
WCWedin 1,148821
if those 'extra' fields are incidental and don't care to do searches on them, I usually go for option 2 (but like JSON better than XML). If there's going to be searches on custom fields, option 3 isn't hard to do, and usually the SQL optimizer can get reasonable performance out of it.
answered Jul 14 '09 at 17:59
Javier 49.6k767112
I am currently working on a project with this same problem, and I have chosen to use option 3, but I added a FieldType field and a ListSource field in case the FieldType="list". The ListSource field could be a query, an sql view, a function name, or something that results in a list of options for the list. The biggest problem with trying to store fields like this in my situation is that this field list can change, and the users are allowed to edit the data later. So what to do if the field list has changed and they go to edit. My solution to that scenario was to allow editing only if the list hasn't changed and to display read-only data if it has.
Avoid stringly-typed data by replacing VALUE with NUMBER_VALUE, DATE_VALUE, STRING_VALUE. Those three types are good enough most of the time. You can add XMLTYPE and other fancy columns later if they're needed. And for Oracle, use VARCHAR2 instead of CHAR to conserve space.
Always try to store values as the correct type. Native data types are faster, smaller, easier to use, and safer.
Oracle has a generic data type system (ANYTYPE, ANYDATA, and ANYDATASET), but those types are difficult to use and should be avoided in most cases.
Architects often think using a single field for all data makes things easier. It makes it easier to generate pretty pictures of the data model but it makes everything else more difficult. Consider these issues:
You cannot do anything interesting with data without knowing the type. Even to display data it's useful to know the type to justify the text. In 99.9% of all use cases it will be obvious to the user which of the 3 columns is relevant. Developing type-safe queries against stringly-typed data is painful. For example, let's say you want to find "Date of Birth" for people born in this millennium:
select * from ReportFieldValue join ReportField on ReportFieldValue.ReportFieldid = ReportField.id where ReportField.name = 'Date of Birth' and to_date(value, 'YYYY-MM-DD') > date '2000-01-01' Can you spot the bug? The above query is dangerous, even if you stored the date in the correct format, and very few developers know how to properly fix it. Oracle has optimizations that make it difficult to force a specific order of operations. You'll need a query like this to be safe:
select * from ( select ReportFieldValue.*, ReportField.* --ROWNUM ensures type safe by preventing view merging and predicate pushing. ,rownum from ReportFieldValue join ReportField on ReportFieldValue.ReportFieldid = ReportField.id where ReportField.name = 'Date of Birth' ) where to_date(value, 'YYYY-MM-DD') > date '2000-01-01'; You don't want to have to tell every developer to write their queries that way. shareimprove this answer answered Aug 2 '15 at 5:56
Jon Heller 23k34186
EAV - Entity-Attribute-Value Implentation
http://mysql.rjweb.org/doc.php/eav
Table of Contents
The Desires It goes by various Names Bad solution The Problems A Solution But what about the ad hoc queries? Why it Works Details on the BLOB/JSON A quick Example Conclusions Postlog Brought to you by Rick James The Desires Open-ended set of "attributes" (key=value) for each "entity". That is, the list of attributes is not known at development time, and will grow in the future. (This makes one column per attribute impractical.) "ad hoc" queries testing attributes. Attribute values come in different types (numbers, strings, dates, etc.) Scale to lots of entities, yet perform well.
It goes by various Names
EAV -- Entity - Attribute - Value key-value RDF -- This is a flavor of EAV MariaDB has dynamic columns that look something like the solution below. They have the added advantage of being able to index the columns otherwise hidden in the blob. (There are caveats.) 5.7 Has JSON datatype, plus functions to access parts MongoDB, CouchDB -- and others -- Not SQL-based.
Bad solution Table with 3 columns: entity_id, key, value The "value" is a string, or maybe multiple columns depending on datatype or other kludges. a JOIN b ON a.entity=b.entity AND b.key='x' JOIN c ON ... WHERE a.value=... AND b.value=...
The Problems The SELECTs get messy -- multiple JOINs Datatype issues -- It's clumsy to be putting numbers into strings Numbers stored in VARCHAR do not compare 'correctly', especially for range tests. Bulky. Dedupping the Values is clumsy. Inefficient implementation of the K-V table. See the following on how to improve WordPress somewhat: wp_postmeta
A Solution Decide which columns need to be searched/sorted by SQL queries. No, you don't need all the columns to be searchable or sortable. Certain columns are frequently used for selection; identify these. You probably won't use all of them in all queries, but you will use some of them in every query.
The solution uses one table for all the EAV stuff. The columns include the searchable fields plus one TEXT/BLOB. Searchable fields are declared appropriately (INT, TIMESTAMP, etc). The BLOB contains JSON-encoding of all the extra fields.
The table should be ENGINE=InnoDB, hence it should have a PRIMARY KEY. The entitity_id is the 'natural' PK. Add a small number of other indexes (often 'composite') on the searchable fields. PARTITIONing is unlikely to be of any use, unless the Entities should purged after some time. (Example: News Articles)
But what about the ad hoc queries? You have included the most important fields to search on -- date, category, etc. These should filter the data down significantly. When you also need to filter on something more obscure, that will be handled differently. The application code will look at the BLOB for that; more on this later.
Why it Works You are not really going to search on more than a few fields. The disk footprint is smaller; Smaller --> More cacheable --> Faster It needs no JOINs The indexes are useful The one table has one row per entity, and can grow as needed. (EAV needs many rows per entity.) Performance is as good as the indexes you have on the 'searchable fields'. Optionally, you can duplicate the indexed fields in the BLOB. Values missing from 'searchable fields' would need to be NULL (or whatever), and the code would need to deal with such.
Details on the BLOB/JSON Build the extra (or all) key-value pairs in a hash (associative array) in your application. Encode it. COMPRESS it. Insert that string into the BLOB. JSON is recommended, but not mandatory; it is simpler than XML. Other serializations (eg, YAML) could be used. COMPRESS the JSON and put it into a BLOB (or MEDIUMBLOB) instead of a TEXT field. Compression gives about 3x shrinkage. When SELECTing, UNCOMPRESS the blob. Decode the string into a hash. You are now ready to interrogate/display any of the extra fields. If you choose to use the JSON features of MariaDB or 5.7, you will have to forgo the compression feature described. MySQL 5.7.8's JSON native JSON datatype uses a binary format for more efficient access.
For JSON in PHP, see $str = json_encode($hash); and $hash = json_decode($str, true); Also, compress() and uncompress().
A quick Example For example, in Real Estate, num_bedrooms and price are far more useful to split out than has_fireplace or septic_tank. All buyers filter on the first two: only a few buyers filter on fireplaces or have a phobia about septic_tanks. So, the explicit columns would include num_bedrooms TINYINT UNSIGNED and price INT UNSIGNED; the others would be buried in the JSON.
Conclusions Schema is reasonably compact (compression, real datatypes, less redundancy, etc, than EAV) Queries are fast (since you have picked 'good' indexes) Expandable (JSON is happy to have new fields) Compatible (No 3rd party products, just supported products) Range tests work (unlike storing INTs in VARCHARs) (Drawback) Cannot use the non-indexed attributes in WHERE or ORDER BY clauses, must deal with that in the app. (5.7 partially alleviates this.)
Postlog Posted Jan, 2014; Refreshed Feb, 2016.
MariaDB's Dynamic Columns MySQL 5.7's JSON
This looks very promising; I will need to do more research to see how much of this blog is obviated by it: Using MySQL as a Document Store in 5.7 more DocStore discussion EAV Fail
If you insist on EAV, set optimizer_search_depth=1.
import java.util.Arrays;
import lombok.Predicates.Predicate1;
public class Main {
public static void main(String[] args) {
Extensions.filter(Arrays.asList(args), isEmpty());
}
private static Predicate1 isEmpty() {
return new Predicate1() {
public boolean evaluate(String s) {
return (s == null) || (s.isEmpty());
}
};
}
}
Predicates.java ->
/**
* Collection of predicate templates.
*/
@NoArgsConstructor(access = AccessLevel.PRIVATE)
public final class Predicates {
/**
* Represents the method that defines a set of criteria and determines whether the specified object meets those
* criteria.
*
* @param
* The type of the object to compare.
*/
public static abstract class Predicate1 extends Function1 {
public final Boolean apply(T1 t1) {
return evaluate(t1);
}
public abstract boolean evaluate(T1 t1);
}
}
/**
* Creates a predicate which can be used to match a string.
*
* @return The predicate which can be used for matching on a string
* @since 1.8
*/
public Predicate asPredicate() {
return s -> matcher(s).find();
}
Deeplearning4j is the first commercial-grade, open-source, distributed deep-learning library written for
Java and Scala. Integrated with Hadoop and Spark, DL4J is designed to be used in business environments on
distributed GPUs and CPUs. Skymind is its commercial support arm.
Deeplearning4j aims to be cutting-edge plug and play, more convention than configuration, which allows for
fast prototyping for non-researchers. DL4J is customizable at scale. Released under the Apache 2.0 license,
all derivatives of DL4J belong to their authors. DL4J can import neural net models from most major frameworks
via Keras, including TensorFlow, Caffe and Theano, bridging the gap between the Python ecosystem and the
JVM with a cross-team toolkit for data scientists, data engineers and DevOps. Keras is employed as
Deeplearning4j's Python API. Machine learning models are served in production with Skymind's model server.
Apache Open NLP vs NLTK
Qn: We have a spring boot application integrated with Node.js and socket.io chat application , to which
we want to integrate Natural language processing. Not getting any direction on which of these two Apache-OpenNlp
or NLTK would be a better choice for us as both of the frameworks offer the kind of processing we need.
Wrt to the features provided by the frameworks , they both are good. Both have features that we are looking for.
More than how to choose between features , what would suit our architecture better is a perspective I would like..
Any suggestions ?
Ans:
It is tough to answer a question about which product will meet your needs better without know what your needs are.
OpenNLP can perform Tokenization, Sentence Detection, POS tagging, Named entity detection, language detection,
Document classification, Chunking, and Sentence Parsing. It also has lower-level access to maximum entropy and
Naive-bayes classifiers. I use OpenNLP often. NLTK appears to do the same stuff (I don't really use it,
so I can't tell you all its benefits). A small difference is that OpenNLP is java whereas NLTK is python.
So your preference can come into play. Another difference is that NLTK has build in methods for downloading corpora.
If you were a little more specific about what you wanted, people could give you better advice.
answered 2 days ago
HowYaDoing
Ans: I think I have.. I know the features provided for both frameworks.. but which would suit an architecture
with the above mentioned technologies is the guidance I would like
Personally, I really like OpenNLP, and because you just need to add the OpenNLP dependencies to maven +
download models. I think it will integrate easily with a spring boot app. -- I do this very thing, and it is easy.
I am sure that the NLTK community would argue that even though they are python-based, you can integrate it
into a java app. Long story short, I suggest openNLP. But keep in mind, I work with AND ON openNLP,
so I have a pro-OpenNLP bias.
What do you think about using DKPro Core? This framework provide a generic API for many NLP frameworks and
comes as Maven dependency. This means you can exchange any of the frameworks. If you also use domain specific
NLP models by building your own corpus, you can use this project to determine your best-performing
NLP pipeline for your domain. Let me know if you have any questions.
@Schrieveslaach I am not familiar with DKPro Core, but it appears that wraps many different NLP projects for UIMA.
This is very useful if you are creating a UIMA-based solution. But I am not sure that Sharanya is considering UIMA,
which brings it's own issues.
@HowYaDoing, it is true that UIMA adds another architectural layer but DKPro Core unifies all NLP tools through
their wrappers. This means Sharanya can exchange any of these tools. I'm don't know the application domain but
if the application domain is different to news papers (NLP models won't work well), the models and tools can
easily be exchanged with a better working version. Therefore, Sharanya can use the second project to find the
best NLP tools for the application domain.
Java vs Python for NLP is very much a preference or necessity. Depending on the company/projects you'll
need to use one or the other and often there isn't much of a choice unless you're heading a project.
Other than NLTK (www.nltk.org), there are actually other libraries for text
processing in python:
The first place to look would be Stanford's Natural Language Processing Group. All of software that is distributed there is written in Java. All recent distributions require Oracle Java 6+ or OpenJDK 7+. Distribution packages include components for command-line invocation, jar files, a Java API, and source code.
Another great option that you see in a lot of machine learning environments here (general option), is Weka. Weka is a collection of machine learning algorithms for data mining tasks. The algorithms can either be applied directly to a dataset or called from your own Java code. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. It is also well-suited for developing new machine learning schemes.
In terms of Java based libraries and tools, another great one that you might look at is LingPipe. alias-i.com/lingpipe
Generally you'd do these two tasks in the other order:
Do part-of-speech tagging
Run a parser using the POS tags as input
OpenNLP's documentation isn't that thorough and some of it's gotten hard to find due to the switch to apache. Some (potentially slightly out-of-date) tutorials are available in the old SF wiki.
You might want to take a look at the Stanford NLP tools, in particular the Stanford POS Tagger and the Stanford Parser. Both have downloads that include pre-trained model files and they also have demo files in the top-level directory that show how to get started with the API and short shell scripts that show how to use the tools from the command-line.
LingPipe might be another good toolkit to check out. A quick search here will lead you to a number of similar questions with links to other alternatives, too!
Which one should i use? Stanford CoreNLP or The Stanford Parser or The Stanford POS Tagger
It depends on what you want/need to do. CoreNLP includes the other two tools plus other annotators, so if you're just experimenting with different kinds of annotation, CoreNLP would be a good place to start. From this question and your related questions, it sounds like you might benefit from reading more about computational linguistics before you get started with your task. I'd suggest Speech and Language Processing by Jurafsky and Martin: cs.colorado.edu/~martin/slp.html
Stanford-nlp's Bootstrapped Entity Learning:
https://nlp.stanford.edu/software/patternslearning.html
Input: seed sets (that is, dictionaries) of entities for some classes and unlabeled text.
Ouput: More entities belonging to the classes extracted from the text.
Algorithm: bootstrapped pattern-based learning.
learning based java (lbjava)
[[mpt: GOOD FOR LEARNING BUT RESTRICTIVE LICENSE; CAN'T USE]]
* Developed by: The Cognitive Computations Group University of Illinois at Urbana-Champaign
* http://cogcomp.cs.illinois.edu/
* http://cogcomp.org/page/software_view/LBJava
Learning Based Java is a modeling language for the rapid development of software systems with one or more learned functions, designed for use with the JavaTM programming language. LBJava offers a convenient, declarative syntax for classifier and constraint definition directly in terms of the objects in the programmer's application. With LBJava, the details of feature extraction, learning, model evaluation, and inference are all abstracted away from the programmer, leaving him to reason more directly about his application.
Allows training; defining custom classifiers.
Lots of sample java code @ http://cogcomp.org/page/software_view/LBJava
Takipi: How to Get Started with Java Machine Learning
By Henn Idan (http://blog.takipi.com/author/henn/) - July 7, 2016
? 5 min read
? 3 comments (http://blog.takipi.com/how-to-get-started-with-javamachine-learning/#disqus_thread)
What are the best tools to get started with Java machine learning?
They've been around for a while, but these days it feels like everyone is talking
about artificial intelligence and machine learning. Its no longer a secret
reserved to scientists and researchers, with implementations in nearly any new
emerging technology.
TAKIPI BLOG (HTTP://BLOG.TAKIPI.COM/)
10/26/2017 How to Get Started with Java Machine Learning | Takipi Blog
http://blog.takipi.com/how-to-get-started-with-java-machine-learning/ 2/7
In the following post we'll do a quick overview of the main Java machine
learning frameworks, and show how easy it is to get started without
reinventing the wheel and creating your own algorithms from scratch.
New Post: How to Get Started with Java Machine Learning
https://t.co/ohD3zCCA6j (https://t.co/ohD3zCCA6j)
pic.twitter.com/MU9kwhIKJM (https://t.co/MU9kwhIKJM)
Takipi (@takipid) July 7, 2016
(https://twitter.com/takipid/status/751082272326086657)
AI for the people
AI is a wide and cool ??eld that has been around for a while, but always felt a
little bit out of reach and made especially for scientists. If you wanted to create
an AI system, you had to implement the core algorithms on your own and train
them to identify patterns, understand images and process natural language.
The recent buzz and evolvement around this ??eld made it more accessible for
non-researchers. Now you have easy access to the relevant algorithms and
tools. You do have to know what you're doing, but its a lot easier to enhance
your applications with machine learning capabilities.
Getting the machine going
To make things simpler, we decided to highlight 3 projects to help get you started:
1. Deeplearning4J (DL4J) (http://deeplearning4j.org/) Open
source, distributed and commercial-grade deep-learning library for JVM
2. BID Data Project (http://bid2.berkeley.edu/bid-data-project/) A
collection of patterns that enable fast, large-scale machine learning
and data mining
3. Neuroph (http://neuroph.sourceforge.net/index.html) Objectoriented neural network
By the way, we recently published another list of interesting open source
GitHub libraries that caught our attention. Check it out
(http://blog.takipi.com/the-hitchhikers-guide-to-github-13-java-projectsyou-should-try/).
1. DL4J Deep Learning
DL4J is a tool made to assist you in the process of con??guring deep neural
networks which are made of multiple layers. It brings deep learning to the JVM
along with fast prototyping and customization at scale, while focusing on more
convention than con??guration.
This is the tool for those who already have the theory needed to create and use
deep neural networks, but dont want to actualize the algorithms themselves.
You can use it to solve speci??c problems involving massive amounts of data
and customize the neural net properties.
DL4J is written in Java, which makes it compatible with any JVM language such
as Clojure, Scala or Kotlin, and it integrates with Hadoop and Spark.
Possible use cases include rating or recommendation systems (CRM, adtech,
churn prevention), predictive analytics or even fraud detection. If you're
looking for a real-world example, you can check out Rapidminer
(https://rapidminer.com/). It's an open-source data platform that uses DL4J to
streamline predictive analytics processes for their users.
Setting up a new neural network is as easy as creating a new object:
Deeplearning4j.java (https://gist.github.com/HennTakipi/b932b32633d81c967f34d0fd769ef302#file-deeplearning4j-java)
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.iterations(1)
.weightInit(WeightInit.XAVIER)
.activation("relu")
.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
.learningRate(0.05)
// ... other hyperparameters
.backprop(true)
.build();
The BID Data Project is made for those of you who deal with a great amount of
data, and are performance sensitive. This UC Berkeley project is a collection of
hardware, software and design patterns that enable fast and large-scale data mining.
The ??rst library is BIDMach, that holds the records for many common machine
learning problems, on single nodes or clusters. You can use it to manage data
sources, optimize and distribute data over CPUs or GPUs.
It includes many popular machine learning algorithms, and the team is working
on developing distributed Deep Learning networks, graph algorithms and
other models.
The other 2 libraries are BIDMat, a fast matrix algebra library that focuses on
data mining and BIDParse, GPU-accelerated natural language parser. Other
libraries in this project include visualization tools, along with libraries that will
let you run on Spark or even on Android.
BIDMach benchmarks repeatedly show better results than other solutions,
even with a single machine compared to alternatives running on larger
clusters. A full list of benchmarks can be found right here
(https://github.com/BIDData/BIDMach/wiki/Benchmarks).
3. Neuroph
Neuroph is a lightweight Java framework used to develop common neural
network architectures. The framework provides a Java library along with a GUI
tool (called easyNeurons), and you can use it in order to create and train your
very own neural networks in Java programs.
It contains an open source Java library, with a small number of basic classes
which correspond to essential neural network concepts. It's a great stepping
stone if you're just getting started with neural networks, or if you want to know
how they work.
You can try out Neuroph online demo (http://neuroph.sourceforge.net/online_demo.html)
and see how it actually
works. Spoiler alert: the interface looks old and outdated, but you can create
nice things with it. Also, it received the Duke's Choice Award for 2013
(http://neuroph.sourceforge.net/dukes_choice_award_2013.html).
(http://384uqqh5pka2ma24ild282mv.wpengine.netdna-cdn.com/wpcontent/uploads/2016/07/network.png)
Network graph view
What About the Others?
In case these 3 projects are not your cup of tea and youre looking for
something a little different for your project, don't worry. If you search GitHub
for Machine learning (https://github.com/search?
l=Java&q=machine+learning&type=Repositories&utf8=%E2%9C%93)ll find 1,506 Java repositories that might give you the right tool.
For example, an interesting project from Airbnb is aerosolve
(https://github.com/airbnb/aerosolve); A machine learning library designed to be human friendly.
Getting started with a new technology is always a source for trouble. If you will
need some help with your exceptions, be sure to check out the OverOps error
analysis tool (https://www.overops.com/).
Final Thoughts
Every few years there's a new buzz around AI. This time around, it came with
reinforcement in the form of machine learning, data mining, neural networks
and so on and we're all for it. The fact that these libraries are open sourced
means that information and abilities are up for grabs, and all you have to do is
think what can be done with this
General Classification Qns & Notes
Qn on Supervised tag suggestion for documents ('multiclass labelling')
I have thousands of documents with associated tag information. However i also have many documents without tags.
I want to train a model on the documents WITH tags and then apply the trained classifier to the UNTAGGED documents; the classifier will then suggest the most appropriate tags for each UNTAGGED document.
I have done quite a lot of research and there doesn't seem to be a SUPERVISED implementation to document tag classification.
I know NLTK, gensim, word2vec and other libraries will be useful for this problem.
I will be coding the project in Python.
Any help would be greatly appreciated.
python machine-learning nlp text-classification
shareimprove this question
asked Jun 26 at 15:55
pwhc
Supervised document tagging isn't particularly uncommon, though it's usually called "multiclass labelling".
For many methods it's the same as single labelling but you pick the N-best results.
See here for a scikit example on made-up data: scikit-learn.org/stable/auto_examples/plot_multilabel.html polm23 Jun 27 at 4:07
Depending on your actual use-case you might opt for more complex method but for minimum working model do:
1) Prepocessing of documents: tokenize, build vocabulary (NLTK has tools for this)
2) Do bag-of-words encoding per document
3) Train a machine learning model with onehot encoding for outputs. Start from sklearn random forest, logistic regression, SVM.
answered Jun 26 at 17:46
Joonatan Samuel
thanks, i have a minimum working model; bow representation of document text with an SVM learning model.
Any advice on taking it further? more pre-processing (stemming ect...). I was thinking of using WMD
to cluster tagged documents, then apply the model to untagged documents that would then find the most
similar kind of tag documents, therefore giving you your tag.
From here it will begin to depend on your actual use case. But basically my answer is in three parts.
1) preprocessing 2) encoding 3) training model.
Depending on your actual use-case and data the prioritization will differ.
E.g. if you have 100mln documents but at test time don't need to worry about computation work
with neural networks to get model better. If you have little data you might want to look into
better preprocessing / encoding much more.
'm currently working on something similar, besides what @Joonatan Samuel suggested I would
encourage you to do careful preprocessing and considerations.
If you want two or more tags for documents you could train several model : one model per tag.
You need to consider if there will be enough cases for each model (tag)
If you have a lot of tags, you could run into a problem with document-tag cases like above.
Stick to most common tag prediction don't try to predict all tags.
answered Jun 26 at 18:01
Diego Agher
in .NET, I'll usually do what was alluded to by samxli. I load all keywords into a hashtable.
I've done it with up to 120,000 keywords and it works pretty fast.
The .NET hashtable object has a contains([key]) method. So for each word in the article I'll just call:
theHashTable.contains(theWord)
If it does contain the word, I'll index it. Has worked pretty well for me without having to use other frameworks.
I don't know how hashtables work in PHP. You'd have to google that. I think their normal arrays work like hashtables?
The key to using a hashtable is that the keys are indexed for fast searching -- I think they use bTrees,
but someone may correct me on that. If you're not familiar with the btree concept, you might want to look that up.
edited Jun 6 '11 at 14:10
answered Jun 6 '11 at 12:37
Trevor
Given *A FIXED SET OF TAGS* one poss approach seems to be to:
expand the tagList into a Map by getting ALL poss associated words (from, for example, the datamuse API)
get freq. keywords from target
lookup ea keyword; find best 'fit'
"MANUALLY TRAIN" the corpus by adding custom tags to the list which will aid disambiguation
[[note: this is the type of method used by https://www.uclassify.com/; who then picks the result if match > 90%]]
Qn: I want a machine to learn to categorize short texts
I have a ton of short stories about 500 words long and I want to categorize them into one of, let's say, 20 categories:
Entertainment
Food
Music
etc
I can hand-classify a bunch of them, but I want to implement machine learning to guess the categories eventually. What's the best way to approach this? Is there a standard approach to machine learning I should be using? I don't think a decision tree would work well since it's text data...I'm completely new in this field.
Any help would be appreciated, thanks!
Ans:
A naive Bayes will most probably work for you. The method is like this:
Fix a number of categories and get a training data set of (document, category) pairs.
A data vector of your document will be sth like a bag of words. e.g. Take the 100 most common words except words like
"the", "and" and such. Each word gets a fixed component of your data vector (e.g. "food" is position 5).
A feature vector is then an array of booleans, each indicating whether that word came up in the corresponding document.
Training:
For your training set, calculate the probability of every feature and every class:
p(C) = number documents of class C / total number of documents.
Calculate the probability of a feature in a class:
p(F|C) = number of documents of class with given feature (= word "food" is in the text) / number of documents in given class.
Decision:
Given an unclassified document, the probability of it belonging to class C is proportional to
P(C|F1, ..., F500) = P(C) * P(F1|C) * P(F2|C) * ... * P(F500|C).
Pick the C that maximizes this term.
Since multiplication is numerically difficult, you can use the sum of the logs instead,
which is maximized at the same C:
log P(C|F1, ..., F500) = log P(C) + log P(F1|C) + log P(F2|C) + ... + log P(F500|C).
Am I right in that the categories of a training set can not be altered? Probably I am a little
bit to lazy to do extended research.
No, they can't. But as soon as you do, you can train a new classifier anyway.
I've classified tens of thousands of short texts. What I did intially was to use a
tf-idf vector space model and then did k-means clustering on those vectors.
This is a very good initial step of exploratory data analysis to get a nice handle on your dataset.
The package I used to cluster was cluto: http://glaros.dtc.umn.edu/gkhome/views/cluto/
To do tf-idf, I just wrote a quick script in perl to tokenize on non-alphanumerics. Then, every
document consists of a bag of words. Every document is represented as a vector of the words it
contains. The value of each index of the vector is the term frequency (tf) * inverse document
frequency (idf). It's just the product of the count of that word/term in the document multiplied
by the reciprocal of the fraction of the documents that contain that word.
(because a word like "the" is very uninformative.)
This method will quickly get you about 80%-90% accuracy. You can then manually label the ones
that are right (or more importantly: wrong) and then do supervised learning if you so choose.
answered Apr 25 '10 at 19:42
user325523
I think the paper "Machine learning in automated text categorization" (you can Google and
download the PDF file) is worth reading. The paper discussed two crucial part:one for feature selection
(translate text to feature space), the other for building a classifier on feature space.
there is a lot of feature selection methods, and several classification methods
(decision tree, naive Bayes, kNN, SVM, etc). you can try some combination to see if
it was working on your data set.
I did something similar before, I use Python for text manipulation, feature selection,
and feature weighting. and Orange for classifier. Orange and Weka already included
naive Bayes, kNN... , but nowadays I might write the classifier with Python script
directly, it shouldn't be very hard too.
Hope this helps.
answered Apr 23 '10 at 7:16
sunqiang
Most people will say that statistical text analysis (like a naive Bayes approach) is
the standard approach: "Foundations of Statistical Natural Language Processing",
Manning and Schuetze and "Speech and Language Processing", Jurafsky and Martin are
the standard references. Statistical text analysis became the standard approach
during the late 90's because they easily outperformed symbolic systems. However
some symbolic systems incorporate statistical elements and you can also actually use
a connectionist approach (there are a few papers demonstrating this). You can also use
cosine similarity (a form of k-Nearest Neighbor) although naive Bayes is usually the
top performer.
Here's a good overview: http://www.cs.utexas.edu/users/hyukcho/classificationAlgorithm.html
I used rainbow mentioned on that page for text classification on a search engine prototype
I wrote on a dot com project.
answered Jan 15 '11 at 16:14
Kirt Undercoffer
Unless there is a chance that you want to do another 500 classifications in the future I
am not sure I would go for a machine learning approach.
Unless the categories are very similar ("food" and "italian food" to take an example) I
think a quite naive heuristic could work very well.
For each category build a table of common words
(for food : "potato", "food", "cook", "tomato", "restaurant",...) and for each text count
which category got the most word matches. Instead of building the dictionary by hand
you could take a sample (say a 100) of the texts, categorize them by hand and then let an
algorithm pick out the words and then make sure to remove words that are common between
all sets (since they provide no information). This is, in essence, a very simple "learning" system.
If you really want a machine learning system there are a number of methods for classification.
The downside is that although most methods are quite simple to implement, the hard part is to
chose a good method, the right features, and good parameters.
answered Apr 23 '10 at 6:12
Hannes Ovr
11.2k34058
This is a good point. Machine learning might not be the best approach for this. Regexps all the way! ash Apr 23 '10 at 9:37
Try Weka... it's a free data mining tool that implements a lot of machine learning algorithms. It has a GUI and an API, so you can use it directly to on your data set or you can program against it.
If you like the results from the various machine learning algorithms and you're still interested in implementing your own algorithms, then you can implement the one(s) that you like the most. This will also help you remove some of the "will it actually work" feeling that you normally get before you build an ML/AI algorithm.
answered Apr 23 '10 at 6:24
Kiril
We can use NLP here. Following would be the steps as I implemented to classify emails in different categories here: 1. Lemmatization: This would remove unnecessary details and would convert all the words into their basic forms or root forms. Like, it will convert working into work, running into run, horses into horse etc. We can Stanford Lemmatizer for this purpose. http://stanfordnlp.github.io/CoreNLP/
Wordnet filtering: We can use only those words which are present in Wordnet. I used Java Wordnet Interface for this purpose. Just filter out the words that are not found in wordnet and take rest of the words. http://projects.csail.mit.edu/jwi/
Find synonyms and further synonyms: For each of the above 5 or 6 categories mentioned above, form separate sets containing synonyms of synonyms of these categories. For e.g., form a set that would contain synonyms of Entertainment and then further synonyms of the synonyms of entertainment found. We can increase this set using web crawling as well.
Feed the data: Take all the words after Lemmatization and Wordnet filtering of a particular story and check how many words matches in each category sets. For e.g., if a story contains 100 words, and it matches with 35 words in entertainment category, 40 words with food, 30 words with travel, then it is highly likely to fall under the category of food and hence it would be a food story. I got good results for my email classification using above approach.
answered Nov 30 '16 at 0:44
Ashish Kumar
Qn: Beginner's resources/introductions to classification?
I'd recommend first starting with Support Vector Machines (SVM) for general applied classification tasks. They'll give you state-of-the-art performance, and you don't really need to understand all of the theory behind them to just use the implementation provided by a package like WEKA.
If you have a larger data-set, you might want to try using Random Forests. There's also an implementation of this algorithm in WEKA, and they train much faster on large data. While they're less broadly used than SVMs, their accuracy tends to match or nearly match the accuracy you could get from one.
Q: Difference between classification and clustering in data mining?
A:
In general, in classification you have a set of predefined classes and want to know which class a new object belongs to.
Clustering tries to group a set of objects and find whether there is some relationship between the objects.
In the context of machine learning, classification is supervised learning and clustering is unsupervised learning.
Classification ->
Use Case: Classify known sample into known classes\
Algorithms: Decision Trees, Bayesian classifiers
Data needs: Labelled samples from a set of classes
Naive-Bayes Text Classification Theory Explained
Qn:
I am finding it hard to understand the process of Naive Bayes, and I was wondering if someone could explain it with a simple step by step process in English. I understand it takes comparisons by times occurred as a probability, but I have no idea how the training data is related to the actual dataset.
Please give me an explanation of what role the training set plays. I am giving a very simple example for fruits here, like banana for example
training set---
round-red
round-orange
oblong-yellow
round-red
dataset----
round-red
round-orange
round-red
round-orange
oblong-yellow
round-red
round-orange
oblong-yellow
oblong-yellow
round-red
Ans:
I realize that this is an old question, with an established answer. The reason I'm posting is that is the accepted answer has many elements of k-NN (k-nearest neighbors), a different algorithm.
Both k-NN and NaiveBayes are classification algorithms. Conceptually, k-NN uses the idea of "nearness" to classify new entities. In k-NN 'nearness' is modeled with ideas such as Euclidean Distance or Cosine Distance. By contrast, in NaiveBayes, the concept of 'probability' is used to classify new entities.
Since the question is about Naive Bayes, here's how I'd describe the ideas and steps to someone. I'll try to do it with as few equations and in plain English as much as possible.
First, Conditional Probability & Bayes' Rule
Before someone can understand and appreciate the nuances of Naive Bayes', they need to know a couple of related concepts first, namely, the idea of Conditional Probability, and Bayes' Rule. (If you are familiar with these concepts, skip to the section titled Getting to Naive Bayes')
Conditional Probability in plain English: What is the probability that something will happen, given that something else has already happened.
Let's say that there is some Outcome O. And some Evidence E. From the way these probabilities are defined: The Probability of having both the Outcome O and Evidence E is: (Probability of O occurring) multiplied by the (Prob of E given that O happened)
One Example to understand Conditional Probability:
Let say we have a collection of US Senators. Senators could be Democrats or Republicans. They are also either male or female.
If we select one senator completely randomly, what is the probability that this person is a female Democrat? Conditional Probability can help us answer that.
Probability of (Democrat and Female Senator)= Prob(Senator is Democrat) multiplied by Conditional Probability of Being Female given that they are a Democrat.
P(Democrat & Female) = P(Democrat) * P(Female | Democrat)
We could compute the exact same thing, the reverse way:
P(Democrat & Female) = P(Female) * P(Democrat | Female)
Understanding Bayes Rule
Conceptually, this is a way to go from P(Evidence| Known Outcome) to P(Outcome|Known Evidence). Often, we know how frequently some particular evidence is observed, given a known outcome. We have to use this known fact to compute the reverse, to compute the chance of that outcome happening, given the evidence.
P(Outcome given that we know some Evidence) = P(Evidence given that we know the Outcome) times Prob(Outcome), scaled by the P(Evidence)
The classic example to understand Bayes' Rule:
Probability of Disease D given Test-positive =
Prob(Test is positive|Disease) * P(Disease)
_______________________________________________________________
(scaled by) Prob(Testing Positive, with or without the disease)
Now, all this was just preamble, to get to Naive Bayes.
Getting to Naive Bayes'
So far, we have talked only about one piece of evidence. In reality, we have to predict an outcome given multiple evidence. In that case, the math gets very complicated. To get around that complication, one approach is to 'uncouple' multiple pieces of evidence, and to treat each of piece of evidence as independent. This approach is why this is called naive Bayes.
P(Outcome|Multiple Evidence) =
P(Evidence1|Outcome) * P(Evidence2|outcome) * ... * P(EvidenceN|outcome) * P(Outcome)
scaled by P(Multiple Evidence)
Many people choose to remember this as:
P(Likelihood of Evidence) * Prior prob of outcome
P(outcome|evidence) = _________________________________________________
P(Evidence)
Notice a few things about this equation:
If the Prob(evidence|outcome) is 1, then we are just multiplying by 1.
If the Prob(some particular evidence|outcome) is 0, then the whole prob. becomes 0. If you see contradicting evidence, we can rule out that outcome.
Since we divide everything by P(Evidence), we can even get away without calculating it.
The intuition behind multiplying by the prior is so that we give high probability to more common outcomes, and low probabilities to unlikely outcomes. These are also called base rates and they are a way to scale our predicted probabilities.
How to Apply NaiveBayes to Predict an Outcome?
Just run the formula above for each possible outcome. Since we are trying to classify, each outcome is called a class and it has a class label. Our job is to look at the evidence, to consider how likely it is to be this class or that class, and assign a label to each entity. Again, we take a very simple approach: The class that has the highest probability is declared the "winner" and that class label gets assigned to that combination of evidences.
Fruit Example
Let's try it out on an example to increase our understanding: The OP asked for a 'fruit' identification example.
Let's say that we have data on 1000 pieces of fruit. They happen to be Banana, Orange or some Other Fruit. We know 3 characteristics about each fruit:
Whether it is Long
Whether it is Sweet and
If its color is Yellow.
This is our 'training set.' We will use this to predict the type of any new fruit we encounter.
Type Long | Not Long || Sweet | Not Sweet || Yellow |Not Yellow|Total
___________________________________________________________________
Banana | 400 | 100 || 350 | 150 || 450 | 50 | 500
Orange | 0 | 300 || 150 | 150 || 300 | 0 | 300
Other Fruit | 100 | 100 || 150 | 50 || 50 | 150 | 200
____________________________________________________________________
Total | 500 | 500 || 650 | 350 || 800 | 200 | 1000
___________________________________________________________________
We can pre-compute a lot of things about our fruit collection.
The so-called "Prior" probabilities. (If we didn't know any of the fruit attributes, this would be our guess.) These are our base rates.
P(Banana) = 0.5 (500/1000)
P(Orange) = 0.3
P(Other Fruit) = 0.2
Probability of "Evidence"
p(Long) = 0.5
P(Sweet) = 0.65
P(Yellow) = 0.8
Probability of "Likelihood"
P(Long|Banana) = 0.8
P(Long|Orange) = 0 [Oranges are never long in all the fruit we have seen.]
....
P(Yellow|Other Fruit) = 50/200 = 0.25
P(Not Yellow|Other Fruit) = 0.75
Given a Fruit, how to classify it?
Let's say that we are given the properties of an unknown fruit, and asked to classify it. We are told that the fruit is Long, Sweet and Yellow. Is it a Banana? Is it an Orange? Or Is it some Other Fruit?
We can simply run the numbers for each of the 3 outcomes, one by one. Then we choose the highest probability and 'classify' our unknown fruit as belonging to the class that had the highest probability based on our prior evidence (our 1000 fruit training set):
P(Banana|Long, Sweet and Yellow)
P(Long|Banana) * P(Sweet|Banana) * P(Yellow|Banana) * P(banana)
= _______________________________________________________________
P(Long) * P(Sweet) * P(Yellow)
= 0.8 * 0.7 * 0.9 * 0.5 / P(evidence)
= 0.252 / P(evidence)
P(Orange|Long, Sweet and Yellow) = 0
P(Other Fruit|Long, Sweet and Yellow)
P(Long|Other fruit) * P(Sweet|Other fruit) * P(Yellow|Other fruit) * P(Other Fruit)
= ____________________________________________________________________________________
P(evidence)
= (100/200 * 150/200 * 50/200 * 200/1000) / P(evidence)
= 0.01875 / P(evidence)
By an overwhelming margin (0.252 >> 0.01875), we classify this Sweet/Long/Yellow fruit as likely to be a Banana.
Why is Bayes Classifier so popular?
Look at what it eventually comes down to. Just some counting and multiplication. We can pre-compute all these terms, and so classifying becomes easy, quick and efficient.
Let z = 1 / P(evidence). Now we quickly compute the following three quantities.
P(Banana|evidence) = z * Prob(Banana) * Prob(Evidence1|Banana) * Prob(Evidence2|Banana) ...
P(Orange|Evidence) = z * Prob(Orange) * Prob(Evidence1|Orange) * Prob(Evidence2|Orange) ...
P(Other|Evidence) = z * Prob(Other) * Prob(Evidence1|Other) * Prob(Evidence2|Other) ...
Assign the class label of whichever is the highest number, and you are done.
Despite the name, Naive Bayes turns out to be excellent in certain applications. Text classification is one area where it really shines.
Hope that helps in understanding the concepts behind the Naive Bayes algorithm.
edited Dec 25 '16 at 10:53
JulienD
3,02921743
answered Dec 12 '13 at 23:54
Ram Narasimhan
Thanks for the very clear explanation! Easily one of the better ones floating around the web. Question: since each P(outcome/evidence) is multiplied by 1 / z=p(evidence) (which in the fruit case, means each is essentially the probability based solely on previous evidence), would it be correct to say that z doesn't matter at all for Naive Bayes? Which would thus mean that if, say, one ran into a long/sweet/yellow fruit that wasn't a banana, it'd be classified incorrectly. covariance Dec 21 '13 at 2:30
@E.Chow Yes, you are correct in that computing z doesn't matter for Naive Bayes. (It is a way to scale the probabilities to be between 0 and 1.) Note that z is product of the probabilities of all the evidence at hand. (It is different from the priors which is the base rate of the classes.) You are correct: If you did find a Long/Sweet/Yellow fruit that is not a banana, NB will classify it incorrectly as a banana, based on this training set. The algorithm is a 'best probabilistic guess based on evidence' and so it will mis-classify on occasion. Ram Narasimhan Dec 21 '13 at 6:35
Ram Narasimhan explained the concept very nicely here below is an alternative explanation through the code example of Naive Bayes in action
It uses an example problem from this book on page 351
This is the data set that we will be using
In the above dataset if we give the hypothesis = {"Age":'<=30', "Income":"medium", "Student":'yes' , "Creadit_Rating":'fair'} then what is the probability that he will buy or will not buy a computer.
The code below exactly answers that question.
Just create a file called named new_dataset.csv and paste the following content.
Age,Income,Student,Creadit_Rating,Buys_Computer
<=30,high,no,fair,no
<=30,high,no,excellent,no
31-40,high,no,fair,yes
>40,medium,no,fair,yes
>40,low,yes,fair,yes
>40,low,yes,excellent,no
31-40,low,yes,excellent,yes
<=30,medium,no,fair,no
<=30,low,yes,fair,yes
>40,medium,yes,fair,yes
<=30,medium,yes,excellent,yes
31-40,medium,no,excellent,yes
31-40,high,yes,fair,yes
>40,medium,no,excellent,no
Here is the code the comments explains everything we are doing here! [python]
import pandas as pd
import pprint
class Classifier():
data = None
class_attr = None
priori = {}
cp = {}
hypothesis = None
def __init__(self,filename=None, class_attr=None ):
self.data = pd.read_csv(filename, sep=',', header =(0))
self.class_attr = class_attr
'''
probability(class) = How many times it appears in cloumn
__________________________________________
count of all class attribute
'''
def calculate_priori(self):
class_values = list(set(self.data[self.class_attr]))
class_data = list(self.data[self.class_attr])
for i in class_values:
self.priori[i] = class_data.count(i)/float(len(class_data))
print "Priori Values: ", self.priori
'''
Here we calculate the individual probabilites
P(outcome|evidence) = P(Likelihood of Evidence) x Prior prob of outcome
___________________________________________
P(Evidence)
'''
def get_cp(self, attr, attr_type, class_value):
data_attr = list(self.data[attr])
class_data = list(self.data[self.class_attr])
total =1
for i in range(0, len(data_attr)):
if class_data[i] == class_value and data_attr[i] == attr_type:
total+=1
return total/float(class_data.count(class_value))
'''
Here we calculate Likelihood of Evidence and multiple all individual probabilities with priori
(Outcome|Multiple Evidence) = P(Evidence1|Outcome) x P(Evidence2|outcome) x ... x P(EvidenceN|outcome) x P(Outcome)
scaled by P(Multiple Evidence)
'''
def calculate_conditional_probabilities(self, hypothesis):
for i in self.priori:
self.cp[i] = {}
for j in hypothesis:
self.cp[i].update({ hypothesis[j]: self.get_cp(j, hypothesis[j], i)})
print "\nCalculated Conditional Probabilities: \n"
pprint.pprint(self.cp)
def classify(self):
print "Result: "
for i in self.cp:
print i, " ==> ", reduce(lambda x, y: x*y, self.cp[i].values())*self.priori[i]
if __name__ == "__main__":
c = Classifier(filename="new_dataset.csv", class_attr="Buys_Computer" )
c.calculate_priori()
c.hypothesis = {"Age":'<=30', "Income":"medium", "Student":'yes' , "Creadit_Rating":'fair'}
c.calculate_conditional_probabilities(c.hypothesis)
c.classify()
output:
Priori Values: {'yes': 0.6428571428571429, 'no': 0.35714285714285715}
Calculated Conditional Probabilities:
{
'no': {
'<=30': 0.8,
'fair': 0.6,
'medium': 0.6,
'yes': 0.4
},
'yes': {
'<=30': 0.3333333333333333,
'fair': 0.7777777777777778,
'medium': 0.5555555555555556,
'yes': 0.7777777777777778
}
}
Result:
yes ==> 0.0720164609053
no ==> 0.0411428571429
Hope it helps in better understanding the problem
peace
edited Mar 29 '16 at 12:00
answered Mar 29 '16 at 11:15
Sayed Zainul Abideen
437411
add a comment
8
down vote
I try to explain the Bayes rule with an example.
Suppose that you know that 10% of people are smokers. You also know that 90% of smokers are men and 80% of them are above 20 years old.
Now you see someone who is a man and 15 years old. You want to know the chance that he be a smoker:
X = smoker | he is a man and under 20
As you know 10% of people are smokers your initial guess is 10% (prior probability, without knowing anything about the person) but the other evidences (that he is a man and he is 15) can contribute to this probability.
Each evidence may increase or decrease this chance. For example the fact that he is a man may increase the chance that he is a smoker, providing that this percentage (being a man) among non-smokers is lower, for example, 40%. In other words being a man must be a good indicator of being a smoker rather than a non-smoker.
We can show this contribution in another way. For each feature, you need to compare the commonness (probability) of that feature (f) independently against its probability under the given condition. (P(f) vs. P(f | x). For example, if we know that the probability of being a man is 90% in a society and 90% of smokers are also men, then knowing that someone is a man doesn't change anything (10% * (90% / 90%) = 10%). But if men contribute to 40% of the society, but 90% of the smokers, then being a man increases the chance of being an smoker (10% * (90% / 40%) = 22.5% ). In the same way, if the probability of being a man was 95% in a population, then regardless of the fact that the percentage of men among smokers is high (90%)!, the evidence of being a man decreases the chance of being an smoker! (10% * (90% / 95%) = 9.5%).
So we have:
P(X) =
P(smoker)*
(P(being a man | smoker)/P(being a man))*
(P(under 20 | smoker)/ P(under 20))
Notice in this formula we assumed that being a man and being under 20 are independent features so we multiplied them, it means that knowing that someone is under 20 has no effect on guessing that he is man or woman. But it may not be true, for example maybe most adolescence in a society are men...
To use this formula in a classifier
The classifier is given some features (man and under 20) and it must decide if he is an smoker or not. It uses the above formula to find that. To gain the needed probabilities (90%, 10%, 80%...) it uses training set. for example it counts the people in the training set that are smokers and find they contribute 10% of the sample. Then for smokers checks how many of them are men or women .... how many are above 20 or under 20....
edited Nov 11 '16 at 17:04
answered Mar 12 '16 at 16:13
Ahmad
Qn: What is naive about Naive Bayes?
Ans: There's actually a very good example on Wikipedia:
In simple terms, a naive Bayes classifier assumes that the presence (or absence) of a particular feature of a class is unrelated to the presence (or absence) of any other feature, given the class variable. For example, a fruit may be considered to be an apple if it is red, round, and about 4" in diameter. Even if these features depend on each other or upon the existence of the other features, a naive Bayes classifier considers all of these properties to independently contribute to the probability that this fruit is an apple.
Basically, it's "naive" because it makes assumptions that may or may not turn out to be correct.
answered May 16 '12 at 8:35
this.lau_
The wikipedia article explains it correctly, but I disagree that "it makes assumptions that may or may not turn out to be correct". With right amount of training data it does a good job of filtering out the irrelevant parameters. The "naive" part is that is does not consider dependence between the parameters.. and hence may have to look at redundant data. Chip May 16 '12 at 8:50
If your data is composed of a feature vector X = {x1, x2, ... x10} and your class labels Y = {y1, y2, .. y5}. Thus, a Bayes classifier identifies the correct class label as the one that maximizes the following formula :
P(y/X) = P(X/y) * P(y) = P(x1,x2, ... x10/ y) * P(y)
So for, it is still not Naive. However, it is hard to calculate P(x1,x2, ... x10/ Y), so we assume the features to be independent, this is what we call the Naive assumption, hence, we end up with the following formula instead
P(y/X) = P(x1/y) * P(x2/y) * ... P(x10/y) * P(y)
answered Aug 28 '13 at 19:44
gr33ndata
It's called naive because it makes the assumption that all attributes are independent of each other. This assumption is why it's called naive as in lots of real world situations this does not fit. Despite this the classifier works extremely well in lots of real world situations and has comparable performance to neutral networks and SVM's in certain cases (though not all).
answered May 22 '14 at 11:09
user3664740
using Naive Bayes in real life
(A seemingly brilliant) Writer mentions using Naive Bayes in real life (football pool)...
...The beauty of a Bayesian classifier is that it's simplicity itself to implement.
The probabilities run away from you on a computer, because multiplying small
probabilities will get you into the IEEE math error zone so quickly your head
will spin, and your system gets all weirded out, but instead of multiplying
probabilities, just use a little bit of seventh grade math (in my day) and
simply sum their logarithms:
p(x | X) = p(a | x) * p(b | x) * p(c | x) ...
can be re-represented as:
log p(x | X) = log p(a | x) + log p(b | x) + log p(c | x) ...
And this way we (greatly) reduce going off into la-la land for your ALU.
WEKA Related
note: these seem to be all outdated; but anyway:
WEKA FAQ: http://weka.wikispaces.com/Frequently+Asked+Questions
IBM DevWorks Intro (Data mining with WEKA): https://www.ibm.com/developerworks/opensource/library/os-weka1/index.html
Another Intro to WEKA: http://ortho.clmed.ncku.edu.tw/~emba/2006EMBA_MIS/3_16_2006/WekaIntro.pdf
Qn: Learning Weka on the Command Line
I am fairly new to Weka and even more new to Weka on the command line. I find documentation is poor and I am struggling to figure out a few things to do. For example, want to take two .arff files, one for training, one for testing and get an output of predictions for the missing labels in the test data.
How can I do this?
I have this code as a starting block
java -classpath weka.jar weka.classifiers.meta.FilteredClassifier
-t "training_file_with_missing_values.arff"
-T "test_file_with_missing_values.arff"
-F weka.filters.unsupervised.attribute.ReplaceMissingValues -- -c last
-W weka.classifiers.functions.MultilayerPerceptron -- -L 0.3 -M 0.2 -H a
Running that code gives me "Illegal option -c last" and I am not sure why. I am also not going to be using MLP as NN tend to be too slow when I have a few thousand features from the text data. I know how to change it to another classifier though (like NB or libSVM so that is good).
But I am not sure how to add multiple filters in one call as I also need to add the StringToWordVector filter (and possibly the Reorder filter to make the class the last, instead of first attribute).
And then how do I get it actually output me the prediction labels of each class? And then store so those in an arff with the initial data.
machine-learning classification weka
asked Mar 15 '13 at 20:21
Reily Bourne
Weka is not really the shining example of documentation, but you can still find valuable information about it on their sites. You should start with the Primer. I understand that you want to classify text files, so you should also have a look at Text categorization with WEKA.
The command line you posted in your question contains an error. I know, you copied it from my answer to another question, but I also just noticed it. You have to omit the -- -c last, because the ReplaceMissingValue filter doesn't like it.
In the Primer it says:
weka.filters.supervised
Classes below weka.filters.supervised in the class hierarchy are for supervised filtering, i.e. taking advantage of the class information. A class must be assigned via -c, for WEKA default behaviour use -c last.
but ReplaceMissingValue is an unsupervised filter, as is StringToWordVector.
Multiple filters
Adding multiple filter is also no problem, that is what the MultiFilter is for. The command line can get a bit messy, though: (I chose RandomForest here, because it is a lot faster than NN).
java -classpath weka.jar weka.classifiers.meta.FilteredClassifier \
-t ~/weka-3-7-9/data/ReutersCorn-train.arff \
-T ~/weka-3-7-9/data/ReutersCorn-test.arff \
-F "weka.filters.MultiFilter \
-F weka.filters.unsupervised.attribute.StringToWordVector \
-F weka.filters.unsupervised.attribute.Standardize" \
-W weka.classifiers.trees.RandomForest -- -I 100 \
Making predictions
Here is what the Primer says about getting the prediction:
However, if more detailed information about the classifier's predictions are necessary, -p # outputs just the predictions for each test instance, along with a range of one-based attribute ids (0 for none).
It is a good convention to put those general options like -p 0 directly after the class you're calling, so the command line would be
java -classpath weka.jar weka.classifiers.meta.FilteredClassifier \
-t ~/weka-3-7-9/data/ReutersCorn-train.arff \
-T ~/weka-3-7-9/data/ReutersCorn-test.arff \
-p 0 \
-F "weka.filters.MultiFilter \
-F weka.filters.unsupervised.attribute.StringToWordVector \
-F weka.filters.unsupervised.attribute.Standardize" \
-W weka.classifiers.trees.RandomForest -- -I 100 \
Structure of WEKA classifiers/filters
But as you can see, WEKA can get very complicated when calling it from the command line. This is due to the tree structure of WEKA classifiers and filters. Though you can run only one classifier/filter per command line, it can be structured as complex as you like. For the above command, the structure looks like this:
The FilteredClassifier will initialize a filter on the training data set, filter both training and test data, then train a model on the training data and classify the given test data.
FilteredClassifier
|
+ Filter
|
+ Classifier
If we want multiple filters, we use the MultiFilter, which is only one filter, but it calls multiple others in the order they were given.
FilteredClassifier
|
+ MultiFilter
| |
| + StringToWordVector
| |
| + Standardize
|
+ RandomForest
The hard part of running something like this from the command line is assigning the desired options to the right classes, because often the option names are the same. For example, the -F option is used for the FilteredClassifier and the MultiFilter as well, so I had to use quotes to make it clear which -F belongs to what filter.
In the last line, you see that the option -I 100, which belongs to the RandomForest, can't be appended directly, because then it would be assigned to FilteredClassifier and you will get Illegal options: -I 100. Hence, you have to add -- before it.
Adding predictions to the data files
Adding the predicted class label is also possible, but even more complicated. AFAIK this can't be done in one step, but you have to train and save a model first, then use this one for predicting and assigning new class labels.
Training and saving the model:
java -classpath weka.jar weka.classifiers.meta.FilteredClassifier \
-t ~/weka-3-7-9/data/ReutersCorn-train.arff \
-d rf.model \
-F "weka.filters.MultiFilter \
-F weka.filters.unsupervised.attribute.StringToWordVector \
-F weka.filters.unsupervised.attribute.Standardize" \
-W weka.classifiers.trees.RandomForest -- -I 100 \
This will serialize the model of the trained FilteredClassifier to the file rf.model. The important thing here is that the initialized filter will also be serialized, otherwise the test set wouldn't be compatible after filtering.
Loading the model, making predictions and saving it:
java -classpath weka.jar weka.filters.supervised.attribute.AddClassification \
-serialized rf.model \
-classification \
-remove-old-class \
-i ~/weka-3-7-9/data/ReutersCorn-test.arff \
-o pred.arff \
-c last
Note:
I want to add that before writing this answer here, I had no knowledge about making predictions from the command line whatsoever. I googled your problems and then came to this answer by trial and error ;)
edited Nov 12 '13 at 12:58
answered Mar 16 '13 at 10:43
Sentry
3,30721832
1
This is one of the most helpful 'guides' to get started using WEKA for prediction purposes - Thanks! SPi Nov 11 '13 at 23:37
2
You are an absolute saint. Is there any way to give you more credit than marking up? You deserve 1k points for this. Gthoma2 Apr 26 '14 at 0:58
A better way to do all that you want to use the GUI Explorer. Here is how to do all that you want:
1) Take two separate files for training and testing.
Use 'Open File' under the Preprocess tab to choose your training file. Use 'Supplied Test Set' radio under the Classify tab to choose your test file.
2) Output the predictions for the missing labels.
Use 'More Options' and choose 'Output Predictions' under the Classify tab to see predictions.
3) Use more than one filters
Use 'Filter' under the Preprocess tab to apply as many filters as you want before classifying.
4) Make class the last attribute
This is actually unnecessary. You can choose any attribute to be your class. A class is any attribute that you want the classifier to predict. Use the Nom(Class) dropdown on the Classify tab to choose which attribute is your class.
answered Mar 15 '13 at 20:33
srrvnn
373618
The problem with your answer for (1) is that when I do that I get the error that they are not compatible. This is because they have different features and will not allow me to test them. That's why I believe I need to use the command line to get them to have the same features. Reily Bourne Mar 15 '13 at 22:26
@JoshWeissbock That's the problem, they must have the same features. Or you must filter them in a way that they have the same features. Sentry Mar 16 '13 at 0:23
Qn: Precision/recall for multiclass-multilabel classification
I'm wondering how to calculate precision and recall measures for multiclass multilabel classification, i.e. classification where there are more than two labels, and where each instance can have multiple labels?
machine-learning classification precision-recall
edited Mar 12 '16 at 13:31
user2314737
Well, false would be if you didn't classified correctly, and true where it was correctly classified. Why do you worry about multiple labels? Thomas Jungblut Jan 25 '12 at 17:10
I found a similar question, this might be a duplicate: stackoverflow.com/questions/3856013/ mehaase May 13 '12 at 23:43
The answer is that you have to compute precision and recall for each class, then average them together. E.g. if you classes A, B, and C, then your precision is:
(precision(A) + precision(B) + precision(C)) / 3
Same for recall.
I'm no expert, but this is what I have determined based on the following sources:
https://list.scms.waikato.ac.nz/pipermail/wekalist/2011-March/051575.html http://stats.stackexchange.com/questions/21551/how-to-compute-precision-recall-for-multiclass-multilabel-classification
answered May 13 '12 at 23:46
mehaase
15.3k44257
3
If your data has unbalanced number of labels, this averaging may not reflect the real performance. tashuhka Aug 13 '14 at 14:25
For multi-label classification you have two ways to go First consider the following.
$n$ is the number of examples.
$Y_i$ is the ground truth label assignment of the $i^{th}$ example..
$x_i$ is the $i^{th}$ example.
$h(x_i)$ is the predicted labels for the $i^{th}$ example.
Example based
The metrics are computed in a per datapoint manner. For each predicted label its only its score is computed, and then these scores are aggregated over all the datapoints.
Precision = $\frac{1}{n}\sum_{i=1}^{n}\frac{|Y_{i}\cap h(x_{i})|}{|h(x_{i})|}$ , The ratio of how much of the predicted is correct. The numerator finds how many labels in the predicted vector has common with the ground truth, and the ratio computes, how many of the predicted true labels are actually in the ground truth.
Recall = $\frac{1}{n}\sum_{i=1}^{n}\frac{|Y_{i}\cap h(x_{i})|}{|Y_{i}|}$ , The ratio of how many of the actual labels were predicted. The numerator finds how many labels in the predicted vector has common with the ground truth (as above), then finds the ratio to the number of actual labels, therefore getting what fraction of the actual labels were predicted.
There are other metrics as well.
Label based
Here the things are done labels-wise. For each label the metrics (eg. precision, recall) are computed and then these label-wise metrics are aggregated. Hence, in this case you end up computing the precision/recall for each label over the entire dataset, as you do for a binary classification (as each label has a binary assignment), then aggregate it.
The easy way is to present the general form.
This is just an extension of the standard multi-class equivalent.
Macro averaged $\frac{1}{q}\sum_{j=1}^{q}B(TP_{j},FP_{j},TN_{j},FN_{j})$
Micro averaged $B(\sum_{j=1}^{q}TP_{j},\sum_{j=1}^{q}FP_{j},\sum_{j=1}^{q}TN_{j},\sum_{j=1}^{q}FN_{j})$
Here the $TP_{j},FP_{j},TN_{j},FN_{j}$ are the true positive, false positive, true negative and false negative counts respectively for only the $j^{th}$ label.
Here $B$ stands for any of the confusion-matrix based metric. In your case you would plug in the standard precision and recall formulas. For macro average you pass in the per label count and then sum, for micro average you average the counts first, then apply your metric function.
You might be interested to have a look into the code for the mult-label metrics here , which a part of the package mldr in R. Also you might be interested to look into the Java multi-label library MULAN.
This is a nice paper to get into the different metrics: A Review on Multi-Label Learning Algorithms
edited Sep 9 '16 at 21:42
answered Sep 9 '16 at 21:33
phoxis
Let us assume that we have a 3-class multi classification problem with labels A, B and C.
The first thing to do is to generate a confusion matrix. Note that the values in the diagonal are always the true positives (TP).
Now, to compute recall for label A you can read off the values from the confusion matrix and compute:
= TP_A/(TP_A+FN_A)
= TP_A/(Total gold labels for A)
Now, let us compute precision for label A, you can read off the values from the confusion matrix and compute:
= TP_A/(TP_A+FP_A)
= TP_A/(Total predicted as A)
You just need to do the same for the remaining labels B and C. This applies to any multi-class classification problem.
Here is the full article that talks about how to compute precision and recall for any multi-class classification problem, including examples.
edited Oct 23 '14 at 20:55
honk
Simple averaging will do if the classes are balanced.
Otherwise, recall for each real class needs to be weighted by prevalence of the class, and precision for each predicted label needs to be weighted by the bias (probability) for each label. Either way you get Rand Accuracy.
A more direct way is to make a normalized contingency table (divide by N so table adds up to 1 for each combination of label and class) and add the diagonal to get Rand Accuracy.
But if classes aren't balanced, the bias remains and a chance corrected method such as kappa is more appropriate, or better still ROC analysis or a chance correct measure such as informedness (height above the chance line in ROC).
answered Mar 20 '15 at 7:18
David M W Powers
Qn: How to interpret weka classification?
How can we interpret the classification result in weka using naive bayes?
How is mean, std deviation, weight sum and precision calculated?
How is kappa statistic, mean absolute error, root mean squared error etc calculated?
What is the interpretation of the confusion matrix?
computer-vision classification weka
shareimprove this question
edited May 10 '12 at 20:54
Atilla Ozgur
Ans:
Below is some sample output for a naive Bayes classifier, using 10-fold cross-validation. There's a lot of information there, and what you should focus on depends on your application. I'll explain some of the results below, to get you started.
=== Stratified cross-validation ===
=== Summary ===
Correctly Classified Instances 71 71 %
Incorrectly Classified Instances 29 29 %
Kappa statistic 0.3108
Mean absolute error 0.3333
Root mean squared error 0.4662
Relative absolute error 69.9453 %
Root relative squared error 95.5466 %
Total Number of Instances 100
=== Detailed Accuracy By Class ===
TP Rate FP Rate Precision Recall F-Measure ROC Area Class
0.967 0.692 0.686 0.967 0.803 0.709 0
0.308 0.033 0.857 0.308 0.453 0.708 1
Weighted Avg. 0.71 0.435 0.753 0.71 0.666 0.709
=== Confusion Matrix ===
a b <-- classified as
59 2 | a = 0
27 12 | b = 1
The correctly and incorrectly classified instances show the percentage of test instances that were correctly and incorrectly classified. The raw numbers are shown in the confusion matrix, with a and b representing the class labels. Here there were 100 instances, so the percentages and raw numbers add up, aa + bb = 59 + 12 = 71, ab + ba = 27 + 2 = 29.
The percentage of correctly classified instances is often called accuracy or sample accuracy. It has some disadvantages as a performance estimate (not chance corrected, not sensitive to class distribution), so you'll probably want to look at some of the other numbers. ROC Area, or area under the ROC curve, is my preferred measure.
Kappa is a chance-corrected measure of agreement between the classifications and the true classes. It's calculated by taking the agreement expected by chance away from the observed agreement and dividing by the maximum possible agreement. A value greater than 0 means that your classifier is doing better than chance (it really should be!).
The error rates are used for numeric prediction rather than classification. In numeric prediction, predictions aren't just right or wrong, the error has a magnitude, and these measures reflect that.
Hopefully that will get you started.
edited Nov 18 '11 at 12:45
answered Aug 16 '10 at 0:33
michaeltwofish
--------------------------------------
To elaborate on michaeltwofish's answer, some notes on the remaining values:
TP Rate: rate of true positives (instances correctly classified as a given class)
FP Rate: rate of false positives (instances falsely classified as a given class)
Precision: proportion of instances that are truly of a class divided by the total instances classified as that class
Recall: proportion of instances classified as a given class divided by the actual total in that class (equivalent to TP rate)
F-Measure: A combined measure for precision and recall calculated as 2 * Precision * Recall / (Precision + Recall)
As for the ROC area measurement, I agree with michaeltwofish that this is one of the most important values output by Weka. An "optimal" classifier will have ROC area values approaching 1, with 0.5 being comparable to "random guessing" (similar to a Kappa statistic of 0).
It should be noted that the "balance" of the data set needs to be taken into account when interpreting results. Unbalanced data sets in which a disproportionately large amount of instances belong to a certain class may lead to high accuracy rates even though the classifier may not necessarily be particularly good.
Further reading:
https://www.cs.auckland.ac.nz/courses/compsci367s1c/tutorials/IntroductionToWeka.pdf
http://en.wikipedia.org/wiki/Receiver_operating_characteristic#Basic_concept
http://en.wikipedia.org/wiki/Information_retrieval#Performance_and_correctness_measures
edited Mar 19 '14 at 16:30
projekt-fisch
6411
answered Feb 4 '14 at 11:31
Hybrid System
Qn: How to read the classifier confusion matrix in WEKA
Sorry, I am new to WEKA and just learning.
In my decision tree (J48) classifier output, there is a confusion Matrix:
a b <----- classified as
130 8 a = functional
15 150 b = non-functional
How do I read this matrix? What's the difference between a & b?
Also, can anyone explain to me what domain values are?
classification weka decision-tree
shareimprove this question
edited Dec 12 '13 at 17:01
Sam Rad
Ans:
Have you read the wikipedia page on confusion matrices? The text around the matrix is arranged slightly differently in their example (row labels on the left instead of on the right), but you read it just the same.
The row indicates the true class, the column indicates the classifier output. Each entry, then, gives the number of instances of that were classified as . In your example, 15 Bs were (incorrectly) classified as As, 150 Bs were correctly classified as Bs, etc.
As a result, all correct classifications are on the top-left to bottom-right diagonal. Everything off that diagonal is an incorrect classification of some sort.
edited Oct 23 '13 at 16:19
answered Mar 5 '13 at 1:28
Junuxx
I'd put it this way:
The confusion matrix is Weka reporting on how good this J48 model is in terms of what it gets right, and what it gets wrong.
In your data, the target variable was either "functional" or "non-functional;" the right side of the matrix tells you that column "a" is functional, and "b" is non-functional.
The columns tell you how your model classified your samples - it's what the model predicted:
The first column contains all the samples which your model thinks are "a" - 145 of them, total
The second column contains all the samples which your model thinks are "b" - 158 of them
The rows, on the other hand, represent reality:
The first row contains all the samples which really are "a" - 138 of them, total
The second row contains all the samples which really are "b" - 165 of them
Knowing the columns and rows, you can dig into the details:
Top left, 130, are things your model thinks are "a" which really are "a" <- these were correct
Bottom left, 15, are things your model thinks are "a" but which are really "b" <- one kind of error
Top right, 8, are things your model thinks are "b" but which really are "a" <- another kind of error
Bottom right, 150 are things your model thinks are "b" which really are "b"
So top-left and bottom-right of the matrix are showing things your model gets right.
Bottom-left and top-right of the matrix are are showing where your model is confused.
answered Oct 28 '14 at 19:14
Mental Nomad
NLP Links
https://www.uclassify.com/
uClassify is a free machine learning web service where you can easily create and use text classifiers.
We offer generous licenses, you can make 500 free calls per day!
Join 41022 developers and bring machine learning into your project! We provide simple and powerful JSON and XML APIs that allows you to use, create, train private and public classifiers
We love machine learning and so does our community who have created 3824 classifiers! Sentiment, Topics, Language detection, IAB, Mood, Gender, Age and Myers Briggs are some of our most popular and many are available in multiple languages!
http://text-processing.com/
Natural Language Processing APIs and Python NLTK Demos:
Natural Language Text Processing APIs
The Text Processing API supports the following functionality:
Stemming & Lemmatization
Sentiment Analysis
Tagging and Chunk Extraction
Phrase Extraction & Named Entity Recognition
The APIs are currently open & free, but limited. If you'd like higher limits, then signup for the Mashape Text-Processing API. If you have any questions, please checkout the FAQ or the StreamHacker blog.
Python NLTK Demos
You can also see demos of all the API functionality:
Stemming Demo
Sentiment Analysis Demo
Tagging and Chunk Extraction Demo
Tokenization Demo
Mallet
Quick Start http://mallet.cs.umass.edu/quick-start.php
Document Classification Quck Start http://mallet.cs.umass.edu/classification.php
MALLET is a Java toolkit for machine learning applied to natural language. It provides facilities for document classification, information extraction, part-of-speech tagging, noun phrase segmentation, general finite state transducers and classification, and much more---all desgined to be extremely efficient for large data and feature sets. Although quite mature in functionality, documentation is still sparse.
The toolkit is Open Source Software, and is released under the Common Public License.
From the dev guide:
Document Classification Developer's Guide
MALLET provides a simple interface to a large collection of classification algorithms. The examples provided here include some of the common tasks required to add classification techniques to your software.
All classifiers (MaxEnt, NaiveBayes, DecisionTree, etc.) extend the Classifier object. Each type of classifier has its own trainer class, all of which extend the ClassifierTrainer class. In this example, we train a MaxEnt classifier using a list of training instances (for information on creating instance lists, see the data import developer's guide).
Allows saving trained Classifiers; allows using multiple algorythms (Naive B. default, also
has options incl Max Entropy/C45/DecisionTree/others) see http://mallet.cs.umass.edu/classification.php
In addition, MALLET provides tools for evaluating classifiers.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Prof. Andrew McCallum
Computer Science Department
University of Massachusetts Amherst
(guy behind MALLET)
Bow: A Toolkit for Statistical Language Modeling, Text Retrieval, Classification and Clustering
Bow (or libbow) is a library of C code useful for writing statistical text analysis, language modeling and information retrieval programs. The current distribution includes the library, as well as front-ends for document classification (rainbow), document retrieval (arrow) and document clustering (crossbow).
Bow Library Front-Ends
Provided in the library source distribution, there are currently three executable programs based on the library.
Rainbow is an executable program that does document classification. While mostly designed for classification by naive Bayes, it also provides TFIDF/Rocchio, Probabilistic Indexing and K-nearest neighbor.
Arrow is an executable program that does document retrieval. It currently only performs simple TFIDF-based retrieval.
Crossbow is a an executable program that does document clustering (and also classification).
Src code: http://www.cs.cmu.edu/~mccallum/bow/src
The code conforms to the GNU coding standards. It is released under the Library GNU Public License (LGPL).
Snippets of some Mallet classes to @Override
As the process moves thro the pipe; the instance data can change type from 1 data type to another.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
/**Alphabet.java
* A mapping between integers and objects where the mapping in each
* direction is efficient. Integers are assigned consecutively, starting
* at zero, as objects are added to the Alphabet. Objects can not be
* deleted from the Alphabet and thus the integers are never reused.
*
* The most common use of an alphabet is as a dictionary of feature names
* associated with a {@link cc.mallet.types.FeatureVector} in an
* {@link cc.mallet.types.Instance}. In a simple doc classification usage,
* each unique word in a document would be a unique entry in the Alphabet
* with a unique integer associated with it. FeatureVectors rely on
* the integer part of the mapping to efficiently represent the subset of
* the Alphabet present in the FeatureVector.
*
* @see FeatureVector
* @see Instance
* @see cc.mallet.pipe.Pipe
*/
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Text2Classify.java:
while (iterator.hasNext()) {
Instance instance = iterator.next();
Labeling labeling = classifier.classify(instance).getLabeling();
[StringBuilder] output.append(instance.getName());
for (int location = 0; location < labeling.numLocations(); location++) {
output.append("\t" + labeling.labelAtLocation(location));
output.append("\t" + labeling.valueAtLocation(location));
}
out.println(output);
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Text2Vectors.java:
/**
* Convert document files into vectors (a persistent instance list).
ArrayList pipeList = new ArrayList();
// Convert the "target" object into a numeric index
// into a LabelAlphabet.
pipeList.add(new Target2Label());
// Set "data" to the file's contents. "data" is now a String.
pipeList.add( new Input2CharSequence(encoding.value) );
// Tokenize the input: first compile the tokenization pattern
tokenPattern = Pattern.compile(tokenRegex.value);
// Add the tokenizer
pipeList.add(new CharSequence2TokenSequence(tokenPattern));
// So far we have a sequence of Token objects that contain
// String values. Look these up in an alphabet and store integer IDs
// ("features") instead of Strings.
pipeList.add( new TokenSequence2FeatureSequenceWithBigrams() );
// write vector file ObjectOutputStream oos;......
oos.writeObject(instances);
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
public class Instance{
protected Object data; // The input data in digested form, e.g. a FeatureVector
protected Object target; // Output data; often a label associated with the instance
protected Object name; // A readable name of the source, e.g. filename
protected Object source; /* The input in a reproducable form, e.g. enabling re-print of
string w/ POS tags, usually without target information,
e.g. human-readable sourceinformation, such as the original text */
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
package cc.mallet.util;
import cc.mallet.types.*;
import java.io.*;
public class InstanceListPrinter {
public static void main(String[] args) throws Exception {
InstanceList instances = InstanceList.load(new File(Paths.get(System.getProperty("user.dir"), "")));
for (Instance instance: instances) {
System.out.println("Name: " + instance.getName());
System.out.println("Target: " + instance.getTarget());
System.out.println("Data: " + instance.getData());
}
}
}
Mallet docs
MALLET Docs
Note: see "User-provided Labeled Features" in
Document Classification with Expectation Constraints
Quick Start
Once you have downloaded and installed MALLET,
the easiest way to get started is through the mallet script. If you
installed MALLET in the directory ~/Applications/mallet, this script
will be in ~/Applications/mallet/bin. The following instructions
assume that your current working directory is the MALLET directory. To use
the script, specify a command and then some number of options, in this pattern:
bin/mallet [command] --option value --option value ...
Type bin/mallet to get a list of commands, and use the option
--help with any command to get a description of valid options.
Data Import: To load all documents in specified directories into a MALLET data file,
with class labels specified by the directory, use the command
This command will run 10 trials, in which the input data is randomly
split into 90% training instances and 10% testing instances. For each trial,
MALLET trains a MaxEnt classifier and a Naïve Bayes classifier on the
training instances, then prints accuracy results and
a matrix of correct and predicted labels for each classifier.
For more information about training and evaluating classifiers, see
the classification quick start guide.
Document Classification
A classifier is an algorithm that distinguishes between a
fixed set of classes, such as "spam" vs. "non-spam", based on
labeled training examples.
MALLET includes implementations of several classification algorithms,
including Naïve Bayes, Maximum Entropy, and Decision Trees.
In addition, MALLET provides tools for evaluating classifiers.
Training Maximum Entropy document classifiers using Generalized
Expectation Criteria is described in this separate tutorial.
To get started with classification, first load your data into MALLET format
as described in the importing data section.
Training a classifier: To train a classifier on a
MALLET data file called training.mallet, use the command
Currently supported algorithms include MaxEnt, NaiveBayes, C45, DecisionTree and many others.
See the JavaDoc API for the cc.mallet.classify package to see the
complete current list of Trainer classes.
Evaluation: No classifier is perfect, so
it is important to know whether a classifier is
producing good results on data not used in training.
To split a single set of labeled instances into training
and testing lists, you can use a command like this:
This command will random split the data into 90% training instances, which
will be used to train the classifier, and 10% testing instances. MALLET
will use the classifier to predict the class labels of the testing instances,
compare those to the true labels, and report results.
Reporting options:
The default report option is a "confusion matrix" showing, for each
true class label (one per row), the number of instances assigned to each predicted
class label (in the colums). Off-diagonal elements represent
prediction errors.
In addition, you can choose to report other statistics such as
accuracy and F1 (a balance between precision and recall). To report accuracy
for the training data and F1
for the class label "sports" on test data, use the command
Multiple random splits / Cross-validation:
Performing multiple test/train splits can
provide a better view of the performance of a classifier. To
use 10 random 90:10 splits, use the options --training-portion 0.9 --num-trials 10.
To use 10-fold cross-validation use --cross-validation 10.
Comparing multiple algorithms:
It is possible to train more than one classifier for each trial,
by simply providing more than one --trainer option:
Note that this option cannot take more than one argument, so you must
add --trainer [type] for every algorithm.
Applying a Saved Classifier to New Unlabeled Data:
To apply a saved classifier to new unlabeled data, use Csv2Classify (for
one-instance-per-line data) or Text2Classify (for one-instance-per-file data).
bin/mallet classify-file --input data --output - --classifier classifier
Using the above commands, classifications are written to standard output.
Note that the input for these commands is a raw text file, not an imported Mallet file.
This command is designed to be used in "production" mode, where labels are not available.
Importing data
MALLET represents data as lists of "instances".
All MALLET instances include a data object. An instance can also include a name and
(in classification contexts) a label. For example, if the application is guessing
the language of web pages, an instance might consist of a vector of word counts
(data), the URL of the page (name) and the language of the page (label).
There are two primary methods for importing data into MALLET format,
first when the source data consists of many separate files, and second
when the data is contained in a single file, with one instance per line.
One instance per file:
After downloading and building MALLET, change to the
MALLET directory.
Assume that text-only (.txt) versions of English web pages are
in files in a directory called
sample-data/web/en and text-only versions of German pages are in
sample-data/web/de
(download sample data).
Now run this command:
MALLET will use the directory names as labels and the filenames as instance names. Note: make sure you are in the mallet directory, not the mallet/bin directory; otherwise you will get a ClassNotFoundException exception.
One file, one instance per line: Assume the data is in the following format:
[URL] [language] [text of the page...]
After downloading and building MALLET, change to the
MALLET directory and run the following command:
In this case, the first token of each line (whitespace delimited, with optional
comma) becomes the instance name, the second token becomes the label, and all
additional text on the line is interpreted as a sequence of word tokens. Note that
the data in this case will be a vector of feature/value pairs, such that a feature
consists of a distinct word type and the value is the number of times that word occurs
in the text.
There are many additional options to the import-dir and import-file
commands. Add the --help option to either of these commands to get a full list.
Some commonly used options to either command are:
--keep-sequence. This option preserves the document as a sequence of word
features, rather than a vector of word feature counts. Use this option for sequence
labeling tasks. The MALLET topic modeling toolkit also requires feature sequences rather
than feature vectors.
--preserve-case. MALLET by default converts all word features to lowercase.
--remove-stopwords. This option tells MALLET to ignore a standard list
of very common English adverbs, conjunctions, pronouns and prepositions.
There are several other options related to stopword
specification.
--token-regex. MALLET divides documents into tokens using a regular expression.
By default, a token is one or more characters in [A-Za-z]. Better options include:
For non-English text,
a good choice is --token-regex '[\p{L}\p{M}]+', which means
Unicode letters and marks (required for Indic scripts).
MALLET currently does not support Chinese or Japanese word segmentation.
If you would like to include punctuation inside tokens (for example contractions like "don't" and internet addresses), you might use --token-regex '[\p{L}\p{P}]*\p{L}', which means any sequence of letters or punctuation marks that ends in a letter. Note that this will include quotation marks at the beginning of words.
SVMLight format: SVMLight-style data in the format
target feature:value feature:value ...
can be imported with
bin/mallet import-svmlight --input train test --output train.mallet test.mallet
Note that the input and output arguments can take multiple files that are processed
together using the same Pipe. Note that the target and feature
fields can be either indices or strings. If they are indices, note that the indices in the
Mallet alphabets and indices in the file may be different, though the data is equivalent.
Real valued targets are not supported.
Data Import for Java Developers
If you require greater flexibility than the command-line
data import tools offer or you would like to embed MALLET
into a larger application, it is useful to understand
the MALLET data import API. This API is based around two
types of classes: data, in the form of Instance objects
contained in InstanceList objects; and classes that
operate on data, iterators and pipes.
A MALLET Instance consists of four fields, which
can be of any Object type.
Name: This field should contain a value
that identifies this instance, usually a String.
Label: The label is primarily used in
classification applications. It is usually a value
within a finite set of labels, a LabelAlphabet.
Data: Generally a FeatureVector (unordered
feature-value pairs) or a FeatureSequence (an ordered
list of features, for example a sequence of words).
Source: A representation of the original
state of the instance. Often a File object or a
String containing the original, untokenized text. Frequently null.
Instances are usually stored in InstanceList objects, which
inherit from ArrayList. Currently the standard format for
storing MALLET data on disk is serialized InstanceLists.
When importing data, it is common to pass an instance through
a series of processing steps. These steps are represented in
MALLET using the Pipe interface. There are a large number of
Pipes -- see the MALLET javadoc API for details. A typical
import pipeline begins with an Iterator, such as a FileIterator
for data that is in one file per instance or a CsvIterator
for data that is in a single file, which is then passed through
a SerialPipes object that wraps a sequence of Pipes.
The following example code takes the name of a directory as input
and recurses through all sub-directories, loading files that end in
.txt as instances. Sample data
is available for download.
Each instance is passed through a series of pipes,
defined in the buildPipe() method. The label of each instance is taken from
the name of the directory that contains the instance.
package edu.umass.cs.iesl.mimno;
import java.io.*;
import java.util.*;
import java.util.regex.*;
import cc.mallet.pipe.*;
import cc.mallet.pipe.iterator.*;
import cc.mallet.types.*;
public class ImportExample {
Pipe pipe;
public ImportExample() {
pipe = buildPipe();
}
public Pipe buildPipe() {
ArrayList pipeList = new ArrayList();
// Read data from File objects
pipeList.add(new Input2CharSequence("UTF-8"));
// Regular expression for what constitutes a token.
// This pattern includes Unicode letters, Unicode numbers,
// and the underscore character. Alternatives:
// "\\S+" (anything not whitespace)
// "\\w+" ( A-Z, a-z, 0-9, _ )
// "[\\p{L}\\p{N}_]+|[\\p{P}]+" (a group of only letters and numbers OR
// a group of only punctuation marks)
Pattern tokenPattern =
Pattern.compile("[\\p{L}\\p{N}_]+");
// Tokenize raw strings
pipeList.add(new CharSequence2TokenSequence(tokenPattern));
// Normalize all tokens to all lowercase
pipeList.add(new TokenSequenceLowercase());
// Remove stopwords from a standard English stoplist.
// options: [case sensitive] [mark deletions]
pipeList.add(new TokenSequenceRemoveStopwords(false, false));
// Rather than storing tokens as strings, convert
// them to integers by looking them up in an alphabet.
pipeList.add(new TokenSequence2FeatureSequence());
// Do the same thing for the "target" field:
// convert a class label string to a Label object,
// which has an index in a Label alphabet.
pipeList.add(new Target2Label());
// Now convert the sequence of features to a sparse vector,
// mapping feature IDs to counts.
pipeList.add(new FeatureSequence2FeatureVector());
// Print out the features and the label
pipeList.add(new PrintInputAndTarget());
return new SerialPipes(pipeList);
}
public InstanceList readDirectory(File directory) {
return readDirectories(new File[] {directory});
}
public InstanceList readDirectories(File[] directories) {
// Construct a file iterator, starting with the
// specified directories, and recursing through subdirectories.
// The second argument specifies a FileFilter to use to select
// files within a directory.
// The third argument is a Pattern that is applied to the
// filename to produce a class label. In this case, I've
// asked it to use the last directory name in the path.
FileIterator iterator =
new FileIterator(directories,
new TxtFilter(),
FileIterator.LAST_DIRECTORY);
// Construct a new instance list, passing it the pipe
// we want to use to process instances.
InstanceList instances = new InstanceList(pipe);
// Now process each instance provided by the iterator.
instances.addThruPipe(iterator);
return instances;
}
public static void main (String[] args) throws IOException {
ImportExample importer = new ImportExample();
InstanceList instances = importer.readDirectory(new File(args[0]));
instances.save(new File(args[1]));
}
/** This class illustrates how to build a simple file filter */
class TxtFilter implements FileFilter {
/** Test whether the string representation of the file
* ends with the correct extension. Note that {@ref FileIterator}
* will only call this filter if the file is not a directory,
* so we do not need to test that it is a file.
*/
public boolean accept(File file) {
return file.toString().endsWith(".txt");
}
}
}
Document Classification
A classifier is an algorithm that distinguishes between a
fixed set of classes, such as "spam" vs. "non-spam", based on
labeled training examples.
MALLET includes implementations of several classification algorithms,
including Naïve Bayes, Maximum Entropy, and Decision Trees.
In addition, MALLET provides tools for evaluating classifiers.
Training Maximum Entropy document classifiers using Generalized
Expectation Criteria is described in this separate tutorial.
To get started with classification, first load your data into MALLET format
as described in the importing data section.
Training a classifier: To train a classifier on a
MALLET data file called training.mallet, use the command
Currently supported algorithms include MaxEnt, NaiveBayes, C45, DecisionTree and many others.
See the JavaDoc API for the cc.mallet.classify package to see the
complete current list of Trainer classes.
Evaluation: No classifier is perfect, so
it is important to know whether a classifier is
producing good results on data not used in training.
To split a single set of labeled instances into training
and testing lists, you can use a command like this:
This command will random split the data into 90% training instances, which
will be used to train the classifier, and 10% testing instances. MALLET
will use the classifier to predict the class labels of the testing instances,
compare those to the true labels, and report results.
Reporting options:
The default report option is a "confusion matrix" showing, for each
true class label (one per row), the number of instances assigned to each predicted
class label (in the colums). Off-diagonal elements represent
prediction errors.
In addition, you can choose to report other statistics such as
accuracy and F1 (a balance between precision and recall). To report accuracy
for the training data and F1
for the class label "sports" on test data, use the command
Multiple random splits / Cross-validation:
Performing multiple test/train splits can
provide a better view of the performance of a classifier. To
use 10 random 90:10 splits, use the options --training-portion 0.9 --num-trials 10.
To use 10-fold cross-validation use --cross-validation 10.
Comparing multiple algorithms:
It is possible to train more than one classifier for each trial,
by simply providing more than one --trainer option:
Note that this option cannot take more than one argument, so you must
add --trainer [type] for every algorithm.
Applying a Saved Classifier to New Unlabeled Data:
To apply a saved classifier to new unlabeled data, use Csv2Classify (for
one-instance-per-line data) or Text2Classify (for one-instance-per-file data).
bin/mallet classify-file --input data --output - --classifier classifier
Using the above commands, classifications are written to standard output.
Note that the input for these commands is a raw text file, not an imported Mallet file.
This command is designed to be used in "production" mode, where labels are not available.
Tutorial: Training Maximum Entropy document classifiers using Generalized Expectation Criteria
Use Mallet 2.0.7 or greater for this code.
The implementation of GE training of MaxEnt models in pre-2.0.7 versions of Mallet contains a bug that often results in low accuracy when the number of constraints is small. Specifically,
the Gaussian prior was not always being included in the objective function value, which caused problems in numerical optimization.
(Published experiments, i.e. [Druck, Mann, and McCallum 2008], used a different implementation and are not affected by this bug.)
To report problems with this code (including obtaining unexpected results), please contact gdruck@cs.umass.edu.
Document Classification with Expectation Constraints
In this tutorial we describe training maximum entropy document classifiers with
expectation constraints that specify affinities between words and labels.
See [Druck, Mann, and McCallum 2008] for more
information. We assume that the task is classifying baseball and hockey documents and that we have
processed data sets baseball-hockey.train.vectors and
baseball-hockey.test.vectors.
These methods require unlabeled training data. We can hide labels using Vectors2Vectors.
If the data is truly unlabeled, then the easiest way to import it is to assign an arbitrary label to each document, ensuring that each label is used at least once.
Generalized Expectation
Suppose we know a priori that the words baseball and puck are good indicators of labels
baseball and hockey respectively. Specifically, suppose that we estimate that 90% of the
documents in which the word puck occurs should be labeled hockey, and similarly for
baseball. We may specify these constraints in a file as follows.
The number of probabilities must be equal to the number of labels. The feature and label names must match the names in the data and target alphabets exactly.
The following command trains a MaxEnt classifier with the above constraints (assumed to be in file
baseball-hockey.constraints) using Generalized Expectation (GE) (as described in [Druck, Mann, and McCallum 2008]).
We specify the constraints file using
constraintsFile and specify a regularization penalty with gasussianPriorVariance.
By default, the difference between the target and model expectations is penalized using KL divergence (as in [Druck, Mann, and
McCallum 2008]). Instead, we can impose an L2 penalty using the L2 option.
The underlying trainer is cc.mallet.classify.MaxEntGETrainer. New GE constraints and penalties for training MaxEnt models can be defined by implementing
cc.mallet.classify.constraints.ge.MaxEntGEConstraint.
Generalized Expectation with Target Ranges
It is also possible to specify L2 constraints that do not impose a penalty if the model expectation is within some target range.
For example, we can encourage model expectations to be in the range 90-100%.
The underlying trainer is cc.mallet.classify.MaxEntGERangeTrainer. New GE constraints and penalties for training MaxEnt models can be defined by implementing
cc.mallet.classify.constraints.ge.MaxEntGEConstraint.
Posterior Regularization
There is also support for training MaxEnt models with Posterior Regularization (PR)
[Ganchev, Graça, Gillenwater, and Taskar 2010].
The following command trains a MaxEnt classifier using the above constraints (assumed to be in file
baseball-hockey.constraints) with PR for 100 iterations. We specify the constraints file using
constraintsFile and specify a regularization penalty for each step (c.f.
[Bellare, Druck, and McCallum 2009]) with pGasussianPriorVariance and qGaussianPriorVariance.
The underlying trainer is cc.mallet.classify.MaxEntPRTrainer. New PR constraints and penalties for training MaxEnt models can be defined by implementing
cc.mallet.classify.constraints.pr.MaxEntPRConstraint.
Automated Methods for Obtaining Constraints
Below, we discuss machine-assisted methods for obtaining constraints. Note that these methods do not yet support target ranges.
User-provided Labeled Features
Rather than specifying the target expectations directly, we may instead specify "labels" for features, and have these converted into target expectations. Suppose we know that the word puck is associated with hockey, and the word baseball is associated with the label baseball. We may specify these labeled features in a file (baseball-hockey.labeled_features) as follows.
baseball baseball
puck hockey
The general format for a file with labeled features is:
feature_name label_name label_name ...
Vectors2FeatureConstraints can estimate target expectations from a file with labeled features. A simple heuristic for obtaining expectations from labeled features is to uniformly divide constant probability mass among the labels for a feature. By default, 0.9 probability is allocated to the labels for a feature. This estimation method can be specified using heuristic for the targets command option.
The option majority-prob can be used to specify a value other than 0.9. We can use the constraints file baseball-hockey.constraints to perform GE training as above.
Machine-provided Candidate Features
We may obtain a set of candidate features for which constraints may be expressed using
the Latent Dirichlet Allocation (LDA) based method of [Druck, Mann, and McCallum 2008].
The lda-file is a serialized LDA model file. See the topic modeling tutorial for more information. Setting targets to none tells Vectors2FeatureConstraints to output candidate features only. baseball-hockey.features will then contain a list of ten candidate features, one per line.
The above method is unsupervised (i.e. does not look at the true labels). We can also select
candidate features using an "oracle" information gain method (infogain) that looks at the
true labels. (Note that when using true labels obtaining constraints, baseball-hockey.train.vectors, rather than baseball-hockey.unlabeled.vectors, must be used.)
Given a set of candidate features, we may estimate constraints using two methods. The first method is to have the machine label the features (by revealing the true labels and using the method of [Druck, Mann, and McCallum 2008]), and convert these labels into expectations using the same
heuristic as above.
Note that when using heuristictargets, the machine may discard candidate features
in the labeling process (c.f. [Druck, Mann, and McCallum 2008]). However, the machine does not discard
any candidate features when using --targets oracle .
Tips
For GE training, a gaussianPriorVariance of 1 is a reasonable default choice.
For PR training, in our experience large values for qGaussianPriorVariance and small values for pGaussianPriorVariance work best.
The command line interfaces only provide basic functionality. In some cases it may be necessary to tweak the optimization code (by for example setting convergence tolerances or step sizes) in
order to obtain good results.
As a rule of thumb, try to specify a set of constraints that is balanced among labels and covers many documents.
Date
Mod
Desc
Column1
Column2
Column3
30-Dec-19
Updated prnEx to printOut exceptions only once per
line #
Removed extra logMsgs for
compiler ("skipping writing class file...") etc.
28-Dec-19
added ui BufferedImg fns from spx + popupMenu for
paginationBar
@ToDo
added defaultColor tint to
desktopIcons(@ToBeTested)
removed the legacy GraphicsUtils stuff
27-Dec-19
continuing the refactor;
esp. data stuff...
26-Dec-19
removed svr stuff
23-Dec-19
refactoring ->
removed actions (moved useful snippets to disable
menuItems)
added prnEx / prnOut
@ToDo
some work on desktopAnim (left off @ ln 1642)
21-Dec-19
worked on spanTbl
(popDat/expandCollapse etc.) in another file (spxServer.spx)
20-Dec-19
Many updates made on 12.18 lost (monkeys messed
w/clipBd & pasted crap)
another reminder to backup
often; the devil does not sleep.
Re-instated changes from 12.18; incl desktopIco
blur
19-Dec-19
Worked on asyncSockCli/Svr;
some progress.Either the code
snippet
contains major errors OR the monkeyBastas up to
their usu tricks (display
doesn't reflect actual code)
17-Dec-19
Added support for pasting last item from
clipBdHistory (via keybd shortcut)
Added new interfaces
userInt, srcRelated & moved fns into them for org
16-Dec-19
Repl all lstDlg calls w/new buildable calls;
removed earlier ver. of class
Repl outline fn w/version
that uses new listDlg
13-Dec-19
Created new
shell for testing interface buildable independently
Buildable working, stable,
ported into system
First impl used
to load embedded srcFiles from jar; working & stable
12-Dec-19
Created cryptoEd.j
w/all the latest updates.Stable
updated
Fatty.jar by importing all regularly used src files; worked on auto-loading
updates to launcher.j to offer embedded
choices.Stable.
updates to
enStrComp.j to suppress/auto-fill class/pkg names; monkeys jam the compiler,
shelved for now
(note that we ported the functionality into
the menu (internal) so launcher was backported for above changes)
spxTbl stable, showing categs.Last work on wonkyTbl (stable) ported into
cryptoEd.j
11-Dec-19
The monkeys are now jamming
the nettySvr (only allows 1 connection; using Stable netty-provided example)
There's a sm chance one import is missing,
will investigate.
10-Dec-19
The monkeys began jamming
the server work (as usu) so decided to use Netty (as we did jetty for ht
httpSvr)
(the reasoning: get the handlers etc.
working & then repl the server w/a custom impl)
9-Dec-19
worked on asyncSocketSvr/Cli
(contd. from legacy work)
8-Dec-19
spxTbl stable,
spanning, several updates to categRenderer
7-Dec-19
Created new
paginationToolBar; added icons (NOTE: subseq. Monkey removed these; might
have to recover)
6-Dec-19
debugging log (monkeys put in non-ascii chars
etc.); these need to be reset
added
recent updates for clipBoard intercepts incl printScn
10-May-19
added
code for "LoadWithNewKey"
09-May-19
applied
changes and saved (encr) w/new stable version
Date
Mod
Desc
28-Dec-20
spxClient
added helperMethod listOf; some fiddling w/compiler
debugged/prepped copyFileToJar for autoSpxToJar
23-Dec-20
spxClient
added diff; some more shuffledKeys
tot keys/riddles in vault=20
22-Dec-20
GridTester
(note: this is not in the file but in a patch; beaucoup @MBI)
worked a bit on Grid Srch button before realizing that it might be moot (can use filter)
created/added new dvView '/by TgtVer'
added versionMap to consts w/alpha/bet in unicode esc
18-Dec-20
spxClient
removed log (to tkDat)
remmed bookmarksPaintCode for ide
renamed some classes
added keys
16-Dec-20
Grid
reconciled updates from last couple of days; updateed dat w/some addendas
set JTbl horizScrPol
upd priorityRndrr
15-Dec-20
General
mostly worked w/clj & predicates
14-Dec-20
Grid
Some work on dvSwitcher; comboBox
13-Dec-20
Grid
Added SubMod/TgtVer to Grid, cal, dat
some other minor grid updates
An incomplete update to the JButton("Switch View")
11-Dec-20
Grid
updates to mTpl/matcher
utilDlg uses matcherPreds & dClone bypass for sysUpdates
other minor updates to refresh tbl etc.
09-Dec-20
minor updates to importer to fix content fld; add default proj etc.
new cls utilDlg
08-Dec-20
minor update to importer to add recs to tkTbl
01-Dec-20 to 08-Dec-20
Grid
postCovid imports into allData
06Oct to 26Oct-20
As before (Feb06 on) the monkeyBastas began being absolute pests;
could not be productive with their constant deleting local files;
replacing clipBd contents w/crap and general nuisance. What normally
would've taken two days (updating the tkTable w/the ~200 new tasks) dragged
on for weeks until they began preventing saves etc.; so productive work was
again impossible and now we've shifted to research & maintenance.
06-Oct-20
more minor bugfixes; the unid turned out to be a tblModel issue
mTpl testing; working as exp so work continues on dvDef etc.
02-Sept-20
Minor update to mTpl file (flatMap unwrap, errs w/empty); recompiled lib to common jar
01-Sept-20
Minor updates incl fn getRschItms
31-Aug-20
Further work on predSupplier in file postCovidTbl
28-Aug-20
Some updates + early testing w/mTpl
27-Aug-20
Resumed work on mTpl; compiled files using jdk8
Removed all the jar junk; we have better versions in legacy code.
26-Aug-20
Added readJarArch(), readEntry() etc. methods to access prior archives
-- can't run the former locally; monkeys adding spurious chars in srcFile as before
25-Aug-20
Continued working on import Mod
One occurance of a NumFmtErr fixed in new .64 file
Many updates to mTpl file; ready to compile whenever
24-Aug-20
Decided to use update tks instd of sample data
Began version on rebuilding mTpl (for rTpl); (incl in this file @ bottom after divider)
wrote minor fns to read/import updates
21-Aug-20
Began work on swing-filterWindow; predBldr
20-Aug-20
Tested available security (insufficient; the meher.zip file doesn't contain
readable files. We have used this file numerous times before; so either:
- the monkeys are replacing the files on the fly (they've done this before)
- something in the settings (jdk ver? security?) is insuff (however, errMsg?)
10-20Aug-20
Entered taskList (>200 items).
Mar17-Aug08-20
Post-COVID shutdown, i.e., 03/17 to 08/08 was spent planning/taskList (offline)
Feb06-Mar17-20
At this point the monkeyBastas began throwing WAY too many
fake compiler errors and generally began being real pests; it became
impossible to be productive with their antics. Therefore the rest of this
period was spent mostly planning/updating the taskList; with occasional updates,
mostly to the Grid; until even that was halted when the library closed on Mar17 (COVID)
6-Feb-20
further refactoring to remove tries & impl
mTuple usage
Removed ALL of 'em (exc some
commented stuff)
next->remove cls tuple (riddle extends)
refactor cypher (also
upgradeVer? poss AFTER?)
refactor grid
5-Feb-20
(GridTester.spx) This is the
stable gridTester ported to the stable (encapsulated) spxClient
Note: BOTH the grid & the client've been
refactored further; but not stable
So this is essentially the
version frozen into 01.29's client
w/A few caveats: Alltbls moved into mTbl; renamed
treeV to dvT
spxCli menuBar left
_in_situ_ & dataMenu inserted w/updates
NO other changes; so once (mTuple/map) refactoring
is complete;
we shd be able to merge it
into this file w/o many probs.
(spxCli) Repl mTuple w/tested version containing
all missing stuff + lift7
4-Feb-20
Some tinkering w/functional
EitherTree (Either+Tree) but it's practically
useless (redundant) in the spxTbl context; in fact
a little naive like
the chap who inserted an
entire tree in col1 of a table to make a "tree-table".
heh.Typical
freshman; a more exp guy will always snicker @ this
but it works, "i used
an OCR library to create an IDE that reads pictures!!".
hehe.clever.very clever.
3-Feb-20
Began refactoring GridTester
w/encapsulation refactoring but the
monkeyBastas began throwing way too many fake
CompilerErrs,
so forced to postone.
Some further refactoring in this file (untested
because they're throwing
fake errs here too, of
course)
2-Feb-20
(Wkend) reading; mostly functional; some testing
1-Feb-20
(Wkend) Created a shell for
testing/building rowRenderer(s) from supplied preds
31-Jan-20
some further cleanup; many fakeMonkeyBastaErrMsgs
(for instance; renamed class UI since it
looks for a nonExistent class;
probably
existing on the basta's machine.Now
it finds one static method but
not another.Perhaps the bastas are simply effing lazy
and only steal/copy
SOME of the
code at one time.Anyway, effers.)
Changed meth
createDesktopMenu to a fn
Updated desktopIcn builder to new build
Removed a few more tries,
repl w mTuple/map
(eventually we want to get to "there IS no
try")
30-Jan-20
ren curr stuff to
"old" (oldmTuple, oldbuild, _Build, oldFinisher)
in order to refactor w/ new classes/methods while
testing piecemeal
REALLY must save more often,
the effing monkeys are behaving like their innate selves
29-Jan-20
created new class Constants; moved static global
stuff there.
further work on incr
encapsulation; removing global refs; removing static
28-Jan-20
NOTE: 3:02pm commit has the latest/stable/tested
mTuple w/Generic Types
backported this file to
01.24 stable; applied appropriate deltas
The last couple of days were spent updating stuff
and mapping over
optionals; which won't do
(no try; we need to map over mTuple)
so the plan is now to
- make encapsulation cleaner
by just passing the ide ref in constr.s
- make mTuple generic, test & THEN update the
build stmts etc.
27-Jan-20
Further refactoring; worked
on removing fn.getTA (repl w/getTab.tA)
as a part of incr encapsulation (removing global
refs incl ide)
updated Outline func class/int regexp to incl
extends/Implements info
further work on fx
Added new _Build() to buildable interface; &
naturally ->
@ToDo
@monkeyBastaInterference;
remmed/postponed (search for "Openv2");
The monkeys are now effing w/the class regexp
Pattern (they're weak at regexp)
@ToDo
this is postponed too; the
debug stmts don't print
@monkeyBastaInterference
Removed beaucoup unnecc
meths from textLN, resulting in
@ToDo
(probably monkeyBastaInduced) reset of tlnPanel
width to 1 digit
updates to desktopIcon.bld
(buildable)
icons not showing (poss @monkeyBastaInt) so added
alt way of launching ide
7-Jan-20
inc spxDlg images from
legacy jarFile
updated Outline labels to all grey (except names)
some further work on ui.dlgs
(interrupted; look for ->
@ToDo
//@monkeyBastaInterference; postponed @0107
6-Jan-20
@monkeyBastaInterference;
lost all deltas; spent the rest of the day reading functional articles
5-Jan-20
reCommented exit calls for badPaddingEx et.al in
cypher
updated Outline func regexps
to highlight ONLY names in color
@ToDo
ported ui classes (spxDlg & related) from spx.j
into this file
Also ported chkAuth which
invokes above (for testing)
<<the animation code was ported from
dlgs.j_08.05>>
@ToDo
Also ported jarFileRelated
fns for ideAutoSave impl.
4-Jan-20
backported desktopIcon img changes to stable ver
(2 deltas; marked @remmed
01.04)
Built new fatty.jar & recompiled launcher for
options;
resulting in various crashes
poss related to @monkeyBastas
trying to launch their add-ons/agents/etc.(a little slow, them)
3-Jan-20
Ported mTuple (from
spxSvr);
added interface fx using mTuple
added cls toaster
Modified regexp for methods in fnOutline (tested;
Stable)
Date
Mod
Desc
03 to 15-Feb-21
general
NoSQL rsch/eval
02-Feb-21
general
rsch/eval: mongoDb, GOOG FireStore
01-Feb-21
general
Some tinkering w/SecurityManager (Jar FileSystem throws), snippet saved
29-Jan-21
general
Composing fns/mapping made me think of routing; and tubelight(!):
mTpl.pointCut (Stable/works; also incl inlined tpl) pastebin jyubUJ2z
28-Jan-21
spxClient
recovered/resaved spxClient.spx with 2-ply qns (Stable) pastebin g1CPFNXq
27-Jan-21
general
The monkeyBastas won't allow the launcher to work.
It's obvious they've replaced the binary (they delete the class file if recompiled)
Related note: they keep retrograding little snippets of code here and there: the compile&Load now reverts to NO fileExt; which was fixed ages ago.
Spent the day reading/research.
26-Jan-21
general
The monkeyBastas continue to test patience; they jam EVERYthing now
Since we changed the pwds, they're probably trying to "force" open coding so they can steal code as usual;
anyway it's useless to fools so spent the day creating a dblList (needs to be bldable) for later
20-Jan-21
spxClient
further research on noSQL/dynamoDB/others
19-Jan-21
spxClient
too much @MBI with almost ANY work; so researched noSQL/dynamoDB
14 to 18-Jan-21
spxClient
WF office closed
13-Jan-21
spxClient
completed work on 'patchify'; couldn't test (@MBI)
created genVault.spx: Stable; updated to gen auth qns; expects riddle# as arg; displays map.put
12-Jan-21
spxClient
created 'patchify' fn/menuItem to persist 21 udpates only to dat
created calGen fn; slmost ready; blds from intStream
11-Jan-21
spxClient
resaved newVer again (w/keys; MBI keep corrupting the save)
08-Jan-21
general
More research on Arrows in General & Kiesli arrows in particular;
decided we can do without
08-Jan-21
Grid
Fixed various icon issues; incl pagTBar
(Why no prob earlier? newKeys)
inlined (and made non-static) mTpl
updated keys (remmed non-working)
updated all copyright msgs for '21
07-Jan-21
general
Some research on Arrows in General & Kiesli arrows in particular
07-Jan-21
Grid
Much @MBI w/newClient; applied some further deltas
06-Jan-21
spxClient
Checked/updated/applied new Keys:
Monkeys rewrote the clipboard content again;
so the vault now contains some commented non-working keys
04-Jan-21
Grid
added versionMap to consts (remmed)
updated keys
added byTgtVer dv stuff in popDat
module Constants = #if Hephaestus let delimFil = let rawFil = let Colors9x9Panel = let chkBoxList = let a2z = let titleTxt:string = let mLTxt:string = let testList = let srchKeywords = let windowedList = let α = let β = let ``↠`` = let verMap = let subScriptNumbers:char list = let superScriptNumbers:char list = #endif module Base = let ``⍲`` = #if November let ``⍭`` f:('a -> 'r) -> 'r = #endif let (|>|) f x = let trueFn = let falseFn = let (|?) pred x y = let inline (?) (obj: 'a) (propName: string) : 'b = let inline (?<-) (obj: 'a) (propName: string) (value: 'b) = let ઓપ્ત_ઓર (opt:option<_>) def = let ઓપ્ત_ટૂ_સ્ટ્રીન્ગ (o:option<'a>):string = let isTuple tuple = let merge xs ys = let tibbie s = let curry f a b = let uncurry f (a,b) = let સ્ટ્રી_લેન (str:string) : int = let સ્ટ્રી_કન્ટેનર્સ (str:string) (str2:string) : bool = let સ્ટ્રી_ટ્રીમ (str:string) : string = let સ્ટ્રી_સપ્લીટ (str:string) (c:string) : string[] = let સ્ટ્રી_રિપ્લ (x:string) (y:string) (z:string) = let સ્ટ્રી_બાઇટ (s: string) : byte[] = let સ્ટ્રી_ફ્રોમ_બાઇટ_અરે (bArr:byte[]):string = let સ્ટ્રી_ઇસ_લોઅર (str:string) : bool = let સ્ટ્રી_ઇસ_ડિજીટ (pos:int) (str:string) : bool = let સ્ટ્રી_ટીલ (str:string) (fstIdx:string) : string = let સ્ટ્રી_આફટર (str:string) (fstIdx:string) : string = let સ્ટ્રી_ગેટ_પેડેડ (padLen:int) (s:string) = let strContains (substr:string) (str:string) = let escHTML s = let tick = let tickSignif = #if !Hotel let rand = let getRnum f s = #else let rand = let getRnum f s = let getSeededRnum f s sd = #endif let ટીન x y z = let ટીનtp x y z = let કારલોસ = let બારી f s t = let ભોલે x = let કરાફે f = let ઍસોપ a b = let પ્રાઈજ a b c = let ગ્રુનડીગ a b = let શૂકામ_શૂકામ x y = let rec regReplSingle (srch:string) (r:Regex) (s:string):string = let લી_મેપ_ફોલ્ડ_સ f l = let લી_મેપ_ફોલ્ડ_લ f l = let liBrk2Tpl (s:string) (l:list<string>) = let લી_રેગેકસ_રિપ્લેસ (s:string) (srch:list<string>) (r:list<Regex>) = let લી_સિવાય_ઈન્ડેક્ષ idx l = let લી_શફલ (listIn:list<_>) (listOut:list<_>) (r:option<Random>) = let લી_અન_શફલ (listIn:list<_>) (listOut:list<_>) (r:option<Random>) = let લી_સિવાય (li:list<_>) (excludeLi:list<_>) = let flattenOpts (li:option<'t> list) : list<'t> = let લી_સિવાય_ઈનડેકસ_લી idxLi l = let throwSpxErr t = let errDivZero() = let tryParseInt (str:string) = let logHR = let fCoreVer = let stat str x = let stati str x = let log x = let db s = let dbx s = let stripQuotes (s: string) : string = let currentDate = let tod = let defDt = let toOb x = let tryOf f x = #if Trump #endif let tryOfL f li = let inline getString (sb : System.Text.StringBuilder) = let inline appendString (sb : System.Text.StringBuilder) (str : string) = let buildString strings = let htmlUnesc s:string = let b64Enc (str : string) = let b64EncB (bArr : byte[]) = let b64Dec (str : string) = let bytesToZipFileBase (x:byte[]) e z = #if Hotel let bytesToZipFile (x:byte[]) e z = let bytesToZip64 x = let testZip() = #endif let (|Even|Odd|) input = let writeC C (s:string) = let toOption x = let tryWith f x = let bi = let serBase x = let ser x = let serToFile x fn = let deSerBase (str : string) = let deSer (str : string) = let deSerToList (fn:string):list<string> = let (|ParseRegex|_|) regex str = let (|BarDelim|_|) (str: string) = let (|TabDelim|_|) (str: string) = #if BushSr let ThreeKingdoms x y z = let Yuan = let જિન x y z = let NorthernAndSouthernDynasties x y z = let સૂઈ x y = let હાન x y z = let ટેન્ગ x y z = let સૂંગ x y z = let મિંગ x y z = let ક્વિન્ગ x y z = let tkEx :seq<int*int> = let વરેપ1 x y z = let વરેપ2 (x,(a,y)) = let વરેપ2a (x,(a,y)) = let વચલૂં mid aft x = let વરેપ befFn mainFn aftFn x = let વરેપ_2 befFn mainFn aftFn x = #endif module Types = let currentDate = let getQn r = let getK r = let newD(id,cr,md,cont) = let datToSStr (d:Dat) = let newAdDoc(id, cr, tit, tags, cont) = let taskToSStr (t:TaskDoc) = let newA(id, cr, tit, tags, cont) = let adToSStr (a:AdDoc) = #if !Papa #else let newTaskImp(s:string) = let newAdImp(s:string) = #endif let intComparer a b = let resultInt = let ex = let Imp x = let Urg x = let getQuad (a:int*int) = let QuadComparer a b = #if Victor #endif let getGenericTypArg (x:obj) = let tkTblDef():TblDefDoc<TaskDoc> = let DesDocInf() = let tkPred = module Session = let initSession (userNm:option<string>) = let cliHandlr (req:string) (g:DataGridView) = module Mtpl = let setInitialEnv (m:Mtpl<'m,'t,'p>) : Mtpl<'r,'e,'s> = let tmpEnv = let empty = let MtplOf x y z = let ``☞`` = let addErr errStr m = let inline (>!>) (f:Mtpl<'m,'t,'p> -> Mtpl<'r,'e,'t>) m = let inline (>!!>) mapList mapper m = let hasFst m = let setFst x m = let getFst m = let hasSnd m = let setSnd x m = let getSnd m = let hasThd m = let setThd x m = let getThd m = let setErr x m = let setEnv x m = module State = let inject m s = let get = let gets f s = let put s = let eval m s = let run m s = let modify f s = let bind f m = let ``⍒`` = let લાય = let લે = let ભગાય = #if Bacchus let ⍵ = #endif let પહેલું_લે x = let બીજું_લે x = let ત્રીજું_લે x = let પહેલું_લાય = let બીજું_લાય = let ત્રીજું_લાય = let ઘૂમાય_તેર = let ઘૂમાય_બત્રિસ = let setM x = let setE x = let getUNID (cls:string) = module Control = let કબૂતર = let પોપટ = let ચકલી = let કાબર = let બુલબુલ = let કાગડો = let ઢેલ = let નિ_કેશ = let નિ_બસ = let સસલૂ = let ખિસખોલી = #if !Otho let modNm2Idx nm = let getK k CombPath mod = #endif module Meta = #if Sierra let getFldInfo(): (Type*string*bool*string) list = let initSetFldTest:TaskDoc = let setFld (o:obj) (fldNm:string) (newVal:obj) = let runTest() = let dTest = let t = let getObVals o = let getEmbObVal o i = let rec recGetFlds o (subT:string option) = let getFlds o = let code1 = let code2 = let synthAssembly = let synthMethod = #endif
CoreExt
as of May 24th:
CoreExt Map
let statTpl msg (m:Mtpl<'m,'t,'p>) : Mtpl<'r,'e,'s> =
let ચકલી_લે x = fun s ->
let પોપટ_લે x = fun s ->
let કબૂતર_લે x = fun s ->
let ચકલી_લાય = fun s ->
let ચકલી_લાય1of2 = fun s ->
let ચકલી_લાય2of2 = fun s ->
let ચકલી_લાય1of3 = fun s ->
let ચકલી_લાય2of3 = fun s ->
let ચકલી_લાય3of3 = fun s ->
let ચકલી_જોડ x = fun s ->
let પોપટ_લાય = fun s ->
let પોપટ_લાય1of2 = fun s ->
let પોપટ_લાય2of2 = fun s ->
let પોપટ_લાય1of3 = fun s ->
let પોપટ_લાય2of3 = fun s ->
let પોપટ_લાય3of3 = fun s ->
let પોપટ_જોડ x = fun s ->
let કબૂતર_લાય = fun s ->
let કબૂતર_લાય1of2 = fun s ->
let કબૂતર_લાય2of2 = fun s ->
let કબૂતર_લાય1of3 = fun s ->
let કબૂતર_લાય2of3 = fun s ->
let કબૂતર_લાય3of3 = fun s ->
let કબૂતર_જોડ x = fun s ->
let વધારે_પડતો_ચૂનો chk msg s =
module WWW = #if Sisyphus let listItr str l = let getNthLitm (lrefs:list<int>) (lget:list<string>) = let getURL1 url = let fetchUrl callback url = let myCallback (reader:IO.StreamReader) url = let getURL (u:string):string = let noose() = #if mjDeck2 let getHeaders = #endif let uri = let client = let content = let AsyncFormPoster() = #endif //Sisyphus
module Base = let getHash (b:byte[]) = let bytJuggler (li:byte[] list) : byte[] list = let isTwoBase = let isThrBase = let getSeed1Of3 = let oneOfTwo dt x y = let getSeed1Of2 = let getPtr() : IntPtr = let iterationCount = let getHdr r sl iv dat = let getVTpl r sl iv dat = let બનારસી_એંક_Base (r:string option) (dat:string) = let બનારસી_એંક_નવું (r:string option) (dat:string) = let બનારસી_એંક_જહૂનૂ (r:string option) (dat:string) = let બનારસી_ડિક (b:string) (p:IntPtr option) = let genVlt() = let genVlt2 (r:string) = let testHeaderDecr() = let commonChk (ans:string) (qL: list<string*string*string*string>) = let removeDupe (li:list<string>) = let testVaultBuilder (i:int) = let asTpl() = #if Obama let testFetch (rtb:RichTextBox) (frm:Form) = #endif #if Oscar let testVlt() = #endif let કલકતી_એંક (inDat:byte[]) (rsaKey:RSAParameters) (oaepPad:bool) = let કલકતી_ડિક (inDat:byte[]) (rsaKey:RSAParameters) (oaepPad:bool) = #if Taft let cspp = let EncrFolder = let DecrFolder = let SrcFolder = let PubKeyFile = let keyName = let CreateAsmKeys = let કલકતી_એંક_ફાઈલ (rsaP:RSAParameters) (inFile:string) = let કલકતી_ડિક_ફાઈલ (rsaP:RSAParameters) (inFile:string) = let ExportPublicKey = let ImportPublicKey = let GetPrivateKey_Click(object sender, EventArgs e) = let (|Banarasi|_|) (પાન: string) = let (|Calcutti|_|) (પાન: string) = let પાન (સુપારી: string) (લવલી:string option) (ક્વિમામ:IntPtr option) (ચબાવો:int option) (પીચાક: bool) = #if Three module ECDH = let bobReceive (encryptedMessage:byte[]) (iv:byte[]) = let alice (bobPubKey: byte[]) = let aliceSend (key:byte[]) (secretMessage:string) = let getBobKey = module testECDH = let CreateKeyPair() = let CreateSharedSecret (privateKey:byte[]) (publicKey:byte[]): byte[] = let aliceKeyPair = let bobKeyPair = let bobSharedSecret:byte[] = let aliceSharedSecret:byte[] = #endif //mjDeck2 #endif //Taft module PathBuilder = #if !Grant let keyGen inL = let postProc aStr = let Rnum() = let getShuffled3 = let extendd li = let extended = let getExtOutcome (mPath2:list<_>) = let scrambled5 (li:string list array) = let scrambled3 (li:string list array) = let scrambled (li:string list array) = let genP inL = let buildPath path sz = let test1() = let bldMask (p1:list<_>) (p2:list<_>) (p3:list<_>) = #endif #if !Coolidge module Cards = let cardSuit = let cardRank = let bldHand (numCrds:int) = #endif module GenTags = let strIn = let initFn (x:int) = let prnL x = let li1 = let genTagsForEa (x:char[]) (y:int) (z:int) : list<string> = let genRandoms() = let testGenTags() = module internal PgnProc = let stripComments (gm:string):string = let getGames (nm: string) (s:string) = let getGroupCounts l = let getLastMove (s:string) = let spawnShellProcess fn gm mv = let getOutp fn gm mv = let runAs l = let procGames l = let runInitialProc (m:Mtpl<'m,'t,'p>) : Mtpl<'r,'e,'s> = let load() = let initPgn = let runStatePGN() = module CodeServices = let getHeader = #if Victor let loadBldInstr doc = let genPortable doc = let loadDocLet = let saveDocLet doc = #endif #if Augustus let uri = let flattenOpts (li:option<'t> list) : list<'t> = let proc (str:string) = let AsyncMain() = #endif let getAdList (key:string):obj = let genDirs =
module Base = let getUNID_Depr (cls:string) : DocUNID = module CitCollector = #if Alfa #if Venom #else #endif #endif module internal DatCreater = module dataReporter = let hdr = let UNIDsList = let TitlesList = let DomsList = let DomsList2 (l:string list) = let AbstractsList = let allTags = let allTags2 l1 l2 l3 = let genContent l1 l2 l3 = let getRpt() = module internal DatImporter = let deSerToList (fn:string):list<string> = let સ્ટ્રી_સપ્લીટ (str:string) (c:string) : string[] = let strUnquote (str:string) = let tkUpdates = let inp = let newTkImp (li:string[]) = let newTkImp2 (s: string) = let pcTaskImporter (l:list<string>) = let adToSStr (a:AdDoc) = let newAdImp (li:string list) = let newAdImp2 (s: string) = let importTasksFromStringDelimFile() = #if Trump #endif #if !Trump let importPCTasks (inp:string) = #else let importPCTasks (inp:string) = #endif let importAdFromStringDelimFile (inp:string) = let getDeSerializedDatFile fn = let runSer inF outF = #if Venom let runSerS() = #endif let listContentGrps datFl = #if Ares let reconcileAdDocs() = #endif #if Alfa let tpl2Type (typ:string) (datFl:string list list) = #endif let listContents fn typ = #if Trump let compareEntries (t1: TaskDoc) (t2: TaskDoc) = let sorted() (tkRes:TaskDoc list) = #endif #if Venom let load (f:string) = let listContains (li1:list<_>) (li2:list<_>) = let genStubs (i:int) = let run = let strUnquote (str:string) = let procLine (str:string) (i:int) = let init() = #endif let importLegacyTasks inp = let imptkUpdItms inp = let (|ParseRegex|_|) r str = let importSnippets inp = let fl() = let loadLegacyTasks() = let loadTkUpdates() = #if Ares let tkLoader = #endif let getZippedTkList outL = let ``લી_સિવાય_ઈનડેકસ_લી`` idxLi l = let applyUpd_1 x = let applyUpd_2 x = let applyUpd_3 x = let applyUpd_4 x = #if Papa let genRpt x = #endif let statGrp hdr li = let listGrps x = let applyTkFilter (x:list<_>) = #if mjDeck3 let tkLoadWithPatches() = #endif #if Sung #endif #if Tang #endif #if Han #endif #if Shan #endif #if Venom let setInitialEnv (m:Mtpl<'m,'t,'p>) : Mtpl<'r,'e,'s> = let mainCE() = #endif let datInit() = let getUnidS (cls:string) = let main = #if mjDeck2 module standAloneDatLoader = let prn s = let strUnquote (str:string) = let prnAdList (ob:list<AdDoc>) = let getDeSerializedDatFile fn = let procLine (str:string) (i:int) = let init() = let legImgs() = let defFont:Font = let rtb = let mainFrm() = let runner() = let prn2 s = let prnAdList2 (ob:list<TaskDoc>) = #if mjDeck2 let byProj (outL:list<TaskDoc>) = #endif #endif module DesignDocs = let colDispIdx:option<list<int>> = let colFillWeight: option<list<float32>> = let colNames = let rowData: list<array<obj>> = let getUNID (cls:string) : DocUNID = let toDojoTableRow tk = let tkTblDef():TblDefDoc<TaskDoc> = let DesDocInf() = let tkPred = let tkDefGrpBy = let tkDefSortBy = let musicDat: list<array<obj>> = let DV1() = let getDeSerTFile fn = let loadAndRun() = let grpBy (projections:list<'T -> 'Key>) (tgtLi:list<'T>) : ('Key * 'T list) list = let inputs = let proj1 = let proj2 = let proj3 = let addTTip (g: DataGridView) (colNum:int) (str:string) : DataGridView = let newD(id,cr,md,cont) = let strBytes (s: string) : byte[] = let dTest = let t = let (TaskDoc(d, pNm, modu, subM, obj, imp, urg, par, comp, dtComp, tgt, docL)) = let (Dat(u, crDt, modDt, tit, cont, tags, flag)) = let (duCaseinfo, duValueArray) = let attribs = let tkFldsLi = let genFldDocs() = #if Diana let isDesTyp (o:obj) = #else let isDesTyp (o:obj) = #endif let desDVSortOrder = let li = let ord = #if Vespasian let SortWithCustomComparer listIn custmOrder = #endif let genDox() =
module Rook = module Trivia = let GetGrkNm (grkNm , romNm , desc , sym) = let GetRomNm (grkNm , romNm , desc , sym) = let GetDesc (grkNm , romNm , desc , sym) = let GetSym (grkNm , romNm , desc , sym) = let GetGndr (grkNm , romNm , desc , sym) = let GetAns g = let ListExcept (li:list<_>) (excludeLi:list<_>) = let getUniqueItems (numNdd:int) (li:list<_>):list<_> = let Rom2Grk = let Grk2Rom = let DescQn = let SymQn = let BeheadedQn = let PrincipateQn = let philoQn = let GenAllOlympianQns = let GenBeheadedQns = let GenPrincipateQns = let pickOneOf4 = let runTrivia (typ:string) = let res(q:string, opts:list<string>, ans:string, inf:string ) = module Setup = let prefixes = let colLi = let numLi = let txt1Li = let txt2Li = let setInitialEnv (m:Mtpl<'m,'t,'p>) : Mtpl<'r,'e,'s> = let runInitSetup () = #if Zeus let applyShuffled3 li = let pickR() = let pickR3 = let rndChk() = #endif #endif //!Minerva #if ump module StateWin = let btnHandler = let initForm = let finalSt = let mainSt = let inp : Mtpl<string,'a list,'b [],'c,'d> = let statState = let inp : Mtpl<Cit list,obj,obj,obj,obj> = #endif module Runner = #if ump let mainCE() = let main argv = #endif let initF () = let defFrm :Form = let main argv =
#nowarn "20" "58" "66" "67" "64" "760" "1182" "1558" module internal Helpers = let defFont:Font = let defPadding = let toP (o:obj):Panel = let toTblP (o:obj):TableLayoutPanel = let toFlowP (o:obj):FlowLayoutPanel = let toBtn (o:obj):Button = let toTxtBox (o:obj):TextBox = let toLbl (o:obj):Label = let titleTxt:string = let titleTxt2:string = let titleTxt3:string = let titleTxt4:string = let mLTxt:string = let testList = let choiceL = let comboChoiceL = let imgLi = let icn = let getAdIcon (tag:string) = let dCobaltBlue:Color = let AirForcefBlue:Color = let LimeBlue:Color = let PlatinumBlue:Color = let MediumDark:Color = let Dark:Color = let Light:Color = let Lighter:Color = let Darkest:Color = let Linen:Color = let SlateGray:Color = let Lightest:Color = let lightColors = let darkColors = let allColors:list<_> = let getRandomColor:Color = let getRandomLightColor:Color = let getRandomDarkColor:Color = let anc = let doc = let ancA = let ancN = let ancTL = let ancLR = let ancTB = let ancTR = let ancBL = let ancBR = let docT = let docB = let docR = let docL = let docN = let docF = let ctrlSnapShot (c: Control) (fn:string) = let getPlatform = let panelAddRange (rng) (p:Panel) : Panel = let ctrlAddRange (rng:Control[]) (c:Control) = let ctrlAddHandler (hdlr:option<EventArgs -> unit>) (c:Control) : Control = let ctrlAddTags (li:list<string*obj>) (c:Control) = let ctrlGetTag (itm:string) (c:Control) = let ctrlGetTagFromTop (tag:string) (o:obj) = let getCurrentRTB (f:Form) = let getAdDoc key = let getImg nm = let txtBtn nm = let getTxtButton nm optF = let getImgButton nm optF = let getChkBoxBtn:CheckBox = let getButton txt imgNm optF :Button = let getLRFlowPanel() = let getTDFlowPanel() = let getTitlePanel = let getTSTxtBox nm :ToolStripItem = let getTSButton txt imgNm optF :ToolStripItem = let getTBItemByNm (strNm:string) (ideTS:ToolStrip):ToolStripItem option = let addDropDn (m:ToolStripMenuItem) (l:list<ToolStripMenuItem>) = let getTxtLbl txt = let getPwdBox = let getMenuItm nm (f:EventArgs -> unit) :ToolStripMenuItem = let dissItm (inO:option<ToolStripMenuItem>) = let dissLi (inO: option<list<ToolStripMenuItem>>) = let addCopyPasteDisableHandlers (c: Control) = let centerOnScreen (d:Form) = let getPnlWith (cList:list<Control>) = let getTblPnlWith (cols:int) (rows:int) (cList:list<Control>) = let getTSImgBtn t i = let gridTSAddRange (li:ToolStripItem list) (ts:ToolStrip) = let StrStatTpnl (tagLi:string list) (d:Form) = let statTpnl (tagLi:string list) (d:Form) = let કબૂતર x = let પોપટ = let ચકલી = let કાબર = let બુલબુલ = let કાગડો = let ઢેલ = let નિ_કેશ = let નિ_બસ = let સસલૂ = let ખિસખોલી = let calcDropDnSize (c:ComboBox) = module DV = let cliHandlr (req:string) (g:DataGridView) = let colDispIdx:option<list<int>> = let colFillWeight: option<list<float32>> = let colNames = let musicDat: list<array<obj>> = let iDat: list<array<obj>> = let getTS (g:DataGridView) = let getToolStrip (g:DataGridView) = let addDragDrop (g:DataGridView) = #if Ares let getToolStripDVList state = let getToolStrip state = let addGridToolBar (f: Form) (g:DataGridView) state = #endif let getCol (g:DataGridView) (n:int):DataGridViewColumn = let addRows (g:DataGridView) (rows:DataGridViewRow[]) = let AddRows (g:DataGridView) (rows:DataGridViewRow[]) = let setColHdrStyle (g: DataGridView) = #if Ares let PopulateDV_v2 (g: DataGridView) (bindingLi:List<Dat>) = let freezeBand (g: DataGridView) (num:int) rowOrcol = #endif let addttip (g: DataGridView) (colNum:int) (str:string) = let addTTip (g: DataGridView) (colNum:int) (str:string) = let populateDV (colDispIdx:option<list<int>>) (colFillWeight:option<list<float32>>) (rowData:list<array<obj>>) (g: DataGridView) = let PopulateDV (dsp:option<list<int>>) (fw:option<list<float32>>) (rowData:list<array<obj>>) (g: DataGridView) = let setupDV (colNames:list<string>) (g:DataGridView) = let SetupDV (colNames:list<string>) (g: DataGridView) = let addChkBoxColAt (i:int) (g:DataGridView) = let addImgColAt (i:int) (g:DataGridView) = let paintCustomImgCell (g:DataGridView) = let getDef (dat:list<array<obj>>) = let setColHdrs (g:DataGridView) = let getPgBar (g:DataGridView) = let getPaginationBar (g:DataGridView) = let categPaintH (g:DataGridView) = let mergeCategRows (g:DataGridView) = #if tbdb #endif #if Ares let loadDef (dat:obj) (g:DataGridView) = let loadDat (dat:obj) (g:DataGridView) = let saveDat (dat:obj) (g:DataGridView) = let dvOpenChk (strDvName:string) (strTblName:string):option DataGridView = let bld (owner:Form) = #endif #if StableVerApr8 let કલકતી_120_પાન (dsp:option<list<int>>) (fw:option<list<float32>>) (colNames:list<string>) (dat:list<array<obj>>) (owner:Form) = let કલકતી_120_પાન (dsp:option<list<int>>) (fw:option<list<float32>>) (colNames:list<string>) (dat:list<array<obj>>) (owner:Form) = #else #endif module Dlg = #if Fillmore let getCInputDlg inpTxt (i:Form) = #endif let getDlgBase (i:Form) (sz: int) = let getDlgBase_v2 (i:Form) (sz: int) = let addTitlePanel (t:string) (dlg:Form) = let addTitlePanel_v2 (t:string) (dlg:Form) = let sizeButtonWids (lb:list<Button>) = let addBtnPanel (n:int) (dlg:Form) = let addBtnPanel_v2 (n:int) (dlg:Form) = let getDlgMini (i:Form) (t:string option) = let getDlg (i:Form) (t:string option) = let getDlg75 (i:Form) (t:string option) = let getDlg100 (i:Form) (t:string option) = let get3BtnDlgMini (i:Form) (t:string option) = let get3BtnDlg (i:Form) (t:string option) = let get3BtnDlg75 (i:Form) (t:string option) = let get3BtnDlg100 (i:Form) (t:string option) = let dlgLayoutHdlr = let dlgLayoutHdlr_v0 = let addPanelAndCustomize (dlg:Form) (midP:Panel) = let addPanelAndCustomize_v2 (dlg:Form) (midP:Panel) = let getDlgMidPanel_v0 (c:Control option) (d:Form) = let getDlgMidPanel (singleCtrl:Control option) (d:Form) = let getTextDlgPanel (strIn:option<string>) (d:Form) = let textConsumeFn (txtIn:string option) (dlg:Form) = let inpConsumeFn (txtIn:string option) (dlg:Form) = #if Hestia let getComboChoiceDlg (befLi:list<string>) (aftLi:list<string>) (d:Form) = let comboChoiceConsumeFn (l:Option<list<string>*list<string>>) (d:Form) = #endif let getListDlgPanel (l:option<list<_>>) (d:Form) = let listConsumeFn (optP:option<_>) (dlg:Form) = let combDlgUpdate (iNm:string) (idxOf:int) (idx:int option) = let getCombDlgPanel (l:option<list<_*string>>) (d:Form) = let CombConsumeFn (optP:option<_>) (dlg:Form) = let getTextDlgPanel_v2 (strIn:option<string>) = let getTextDlgPanel_v3 (strIn:option<string>) = let textConsumeFn_v2 (dlg:Form) = let textConsumeFn_v3 (dlg:Form) = let getSecDlgPanel = let secDlgConsumeFn (dlg:Form) = #if !Foxtrot let getPathDlgPanel (inL:list<string>) = let pathConsumeFn (dlg:Form) = #endif #if Mars let listOrderingConsumeFn (dlg:Form) = let monoTesterForListBox = #endif #if Diana let getTagCloudPanel (tl:list<(string * int)>) = let tagDlgConsumeFn (dlg:Form) = #endif #if !Pierce #else let getAllCardsDlgPanel = #endif #if VanBuren let addCombos (inL:list<string*string*string>) (f: Form) (p:TableLayoutPanel) : TableLayoutPanel = let getArrowBtnPanel (cB:ComboBox) (lB:ListBox) : FlowLayoutPanel = let getPathComboBoxes (inL:list<list<string>>):Panel = let getPathPanel (lB:ComboBox) (brackts:list<string>) = let getPathListDlgBtnPanel = let getPathListDlg (i: Form) (li:list<list<string>>) (brackts:list<string>) = let getPathParams = let bldPath (f:Form) = let bldPathS = let getCloudDlg (dlg: Form) (l:option<list<string*int>>) (t:string) (dlgTyp:int) = let createAndShowMDIFrm = #if Three #endif #endif //Bacchus let toggleViz (d:Form) = let કબૂતર2 = let પલંગ_તોડ_પાન (સુપારી:int) (લવલી:string option) (ક્વિમામ:option<_>) પીચાક (બનાવો:Form) = module Desktop = let getDesktopHandler hdlrNm state = let getStatusBar state = let addDesktopIcons pnl state = let dskPopupEventHandler = let dskLayoutHdlr = let getDesktopForm (li:(Bitmap*string*string*string) list) :Form = let getDskFrm() = let bld() = module Ide = let diffOb = #if !Clinton let getDiffsLineMode (bef:string) (aft:string) = #endif let getDiffs (bef:string) (aft:string) = let getDiffsHtml (bef:string) (aft:string) (frm:Form) = let getPatch (bef:string) (aft:string) (benc:bool) (frm:Form) = let applyPatch patch (bef:string) = let setupMainFrm (frm:Form) = let main argv = #if Minerva #endif let addUIMenu (frm:Form) = #if !Clinton #endif #if !Hestia #endif #if !Hestia #endif let frm = let runner() = module Crypto = #if tbdb let getKeyPadDlgFrm:Form = let keyPadBtnPanel = let keyPadTblPanel = let keyPadDlg() = let get9x9Panel (i:Control) = let getTblDlg (i: Form) (li:list<Riddle>) (o:option<EventArgs -> unit>) = #endif module Form = #if Alcibiades let colorPicker = let fontPicker = let getCommDlgPnl (lblTxt:string) (dlg:Form) (h:EventHandler) : Panel = let bldCfgForm (owner:Form) = let fldList = let getDefaultLbl (t:string):Label = let getDefaultTextBox:TextBox = let getTaskRangeWidget = let getLrgTxtBox = let getDatePicker = let getDefaultFldPanel = let getFldPanel (rng:Control[]) = let getLFldPanel (rng:Control[]) = let getPanels acc x = let build (li:list<string*FldTypes>) = #endif let btnCl (x:int) = let txtVal (x:int) = #if Taft let validatingForm:Form = let errorProviderForm:Form = #endif module Chess = let getGlyph (pc:string): string = let glyphToChar (pc:string): string = let getBdPos (toPl:string) (col:int) (row:int) = let getBdHdrs (toPl:string) = let transformFen (thisRow:string) = let toBoard (li:string list list) = let strIsDigit (str:string) : bool = let getToPlay (fen:string) = let genDat (fen:string) = let addDragDrop (g:DataGridView) = let g() = let assignHdrFmt (g: DataGridView) = let chessDlgGrid() = let assignDat (rowData:string list list) (g: DataGridView) = let assignHdrs toPlay (g: DataGridView) = let chessConsumeFn = let getChessDlg (fenStr:string) (d:Form) = #if Pythogoras module Main = let addUIMenu (frm:Form) = #if Neptune #endif #if Neptune #endif let appContext = let uiTester:Form = let main argv = #endif //Pythagoras
module Core let delimFil let rawFil let Colors9x9Panel let chkBoxList let a2z let srchKeywords let windowedList let T let CJ let ``α`` let ``β`` let ``↠`` let verMap let (|>|) f x let trueFn let falseFn let (|?) pred x y let optOr (opt:option<'a>) def let isTuple tuple let merge xs ys let tibbie s let curry f a b let uncurry f (a,b) let strLen (str:string) : int let strContains (str:string) (str2:string) : bool let strTrim (str:string) : string let strSplit (str:string) (c:string) : string[] let strRepl (x:string) (y:string) (z:string) let strBytes (s: string) : byte[] let strFromBarr (bArr:byte[]):string let throwSpxErr t let errDivZero() let rand let swap (a: _[]) x y let shuffle a let tryParseInt (str:string) let getRnum f s let logHR let stat str x let stati str x let log x let stripQuotes (s: string) : string let currentDate let defDt let toObj x let oToStr (o:option<'t>) let tryOf f x let flattenOpts (li:option<'t> list) : list<'t> let tryOfL f li let inline getString (sb : System.Text.StringBuilder) let inline appendString (sb : System.Text.StringBuilder) (str : string) let buildString strings let htmlUnesc s:string let b64Enc (str : string) let b64EncB (bArr : byte[]) let b64Dec (str : string) # let bytesToZipFile (x:byte[]) e z let bytesToZip64 x let testZip() # let (|Even|Odd|) input let logMsg msg let logMsgN msg let toOption x let tryWith f x let bi let ser x let deSer (str : string) let deSerToList (fn:string):list let (|ParseRegex|_|) regex str let (|BarDelim|_|) (str: string) let (|TabDelim|_|) (str: string) let M let L let O let T let W let W1 let B let H let C let R2 let R let F let V let combine let Q1 let Q let Q2 let Q3 let Q4 # let L let B() let W let C() let G() let HoT() let ML x y # let ThreeKingdoms x y z let Yuan let Jin x y z let NorthernAndSouthernDynasties x y z let Sui x y let addStr (x:string) let addStr' (x:string) let addStr2 (x:string) (y:string) let Han let Tang x y z let Song let Ming x y z let Qing x y z let tkEx :seq let currentDate type SessionId type DocUNID let getUNID (cls:string) : DocUNID type OptModID type DocVersionModule type DocEditHistoryModule type DocAttachmntModule type DocImgModule type DocRelationModule type Riddle let getQn r let getK r type Mpath type TriviaRiddleQn type Dat let newD(id,cr,md,cont) let datToSStr (d:Dat) type Cit type SpxLog type Snippet type AdDoc let newAdDoc(id, cr, tit, tags, cont) type ArticleDoc type TgtVer type TaskDoc let taskToSStr (t:TaskDoc) let newA(id, cr, tit, tags, cont) let adToSStr (a:AdDoc) let newTaskImp(s:string) let newAdImp(s:string) type Quad let intComparer a b let resultInt let ex let Imp x let Urg x let getQuad (a:int*int) let QuadComparer a b type IdeW type DskW type GridW type activeWin type Env type Env let initSession (userNm:option) let cliHandlr (req:string) (g:DataGridView) type Mtpl<'m,'t,'p> let tmpEnv let empty let MtplOf x y z let addErr errStr m let inline (>!>) (f:Mtpl<'m,'t,'p> -> Mtpl<'r,'e,'t>) m let inline (>!!>) mapList mapper m let hasFst m let setFst x m let getFst m let hasSnd m let setSnd x m let getSnd m let hasThd m let setThd x m let getThd m let setErr x m let setEnv x m type State<'St, 'm> let inject m s let get let gets f s let put s let eval m s let run m s let modify f s let bind f m type StateBuilder() let ``⍒`` let setF x let setM x let setE x # type CombPath let modNm2Idx nm let getK k CombPath mod # module Comm # let listItr str l let getNthLitm (lrefs:list) (lget:list) let getURL1 url let fetchUrl callback url let myCallback (reader:IO.StreamReader) url let getURL (u:string):string let google() let getHeaders let uri let client let content let AsyncFormPoster() # module Crypto let getPtr() : IntPtr let iterationCount let getHdr r sl iv dat let AESEnc (r:string option) (dat:string) : string let AESDec (b:string) (p:IntPtr option) let genVlt() # let testFetch (rtb:RichTextBox) (frm:Form) # let testVlt() let RSAEncr (inDat:byte[]) (rsaKey:RSAParameters) (oaepPad:bool) let RSADecr (inDat:byte[]) (rsaKey:RSAParameters) (oaepPad:bool) # let cspp let EncrFolder let DecrFolder let SrcFolder let PubKeyFile let keyName let CreateAsmKeys let EncryptFile rsa inFile let DecryptFile rsa inFile let ExportPublicKey let ImportPublicKey let GetPrivateKey_Click(object sender, EventArgs e) # # let keyGen inL let postProc aStr let Rnum() let getShuffled3 let extendd li let extended let getExtOutcome (mPath2:list<_>) let scrambled5 (li:string list array) let scrambled3 (li:string list array) let scrambled (li:string list array) let genP inL let buildPath path sz let test1() let bldMask (p1:list<_>) (p2:list<_>) (p3:list<_>) # # type Rank type Suit type Card let desc (c:Card) let getRnum f s let bldHand let tryParse (str:string) let testL let resL let runCards() # let strIn let initFn (x:int) let prnL x let li1 let genTagsForEa (x:char[]) (y:int) (z:int) : list let genRandoms() let testGenTags() let rec stripComments (gm:string) (i:option):string let getGames (nm: string) (s:string) let getGroupCounts l let getLastMove (s:string) let spawnShellProcess fn gm mv let getOutp fn gm mv let runAs l let procGames l let runInitialProc (m:Mtpl<'m,'t,'p>) : Mtpl<'r,'e,'s> let load() let initPgn let runStatePGN() module Data # let (|StrOut|_|) (str: string) let (|LrbCit|_|) (str: string) let (|ALDailyCit|_|) (str: string) let (|ParseRegex|_|) regex str let parseValWithActivePattern str let parseVal (str:string) let lrbUrlBase let lrbUrlNums let lrbExpanded let aldUrlBase let aldUrlNums let aldExpanded let runAlDgetter() let uri let autoParse2 tpl let autoParse1 (s:string) let breakTo5 (s:string) let run() # let listRaw() let procDat (dat:string list) (u:string) let tL let toOpt let funcMapFld acc (x:Cit option) let procRes let flattened let res, acc let collapsedRes let procCits() # let tkUpdates let inp let newTaskImp(s:string) let adToSStr (a:AdDoc) let newAdImp(s:string) let importTasksFromStringDelimFile() let importAdFromStringDelimFile(inp:string) let ser x let getDeSerializedDatFile() let runSer() let runSerS() let listContents() let reconcileAdDocs() let init() let testStateVersion() # module UI let defFont:Font let ctrlAddRange (rng:Control[]) (c:Control) : Control let ctrlAddHandler (hdlr:option unit>) (c:Control) : Control let getAdDoc key let getImg nm let txtBtn nm let getTxtButton nm optF let getImgButton nm optF let getChkBoxBtn:CheckBox let getButton txt imgNm optF :Button let getTSTxtBox nm :ToolStripItem let getTSButton txt imgNm optF :ToolStripItem let getTxtLbl txt let getPwdBox let getMenuItm nm (f:EventArgs -> unit) :ToolStripMenuItem let addCopyPasteDisableHandlers (c: Control) let centerFrm (f:Form) let getDefaultDlgFrm (i:Form) let getInpDlg msg (i:Form) let getChoiceDlg msg (i:Form) (l1:list) (l2:list) let getCInputDlg inpTxt (i:Form) let getTitlePanel str let getLRFlowPanel let getTDFlowPanel let getDefaultDlgBtnPanel (o:option unit>) # let getBtnPanel let addMenu (frm:Form) # let getTxtDlg (i: Form) (strIn:option) (o:option unit>) let get9x9Panel (i:Control) let getTblDlg (i: Form) (li:list) (o:option unit>) # let getDlgFrm:Form let btnPanel let titlePanel s let btnCl (x:int) let txtVal (x:int) let tblPanel let dlg1() let validateFrm (sender:obj) (evtArgs:EventArgs) let validatingForm() let errorProviderForm:Form # let getHeader # let loadBldInstr doc let genPortable doc let loadDocLet let saveDocLet doc # # let uri let flattenOpts (li:option<'t> list) : list<'t> let proc (str:string) let AsyncMain() # # type sDlg let addCombos (inL:list) (f: Form) (p:TableLayoutPanel) : TableLayoutPanel let getArrowBtnPanel (cB:ComboBox) (lB:ListBox) : FlowLayoutPanel let getPathComboBoxes (inL:list>):Panel let getPathPanel (lB:ComboBox) (brackts:list) let getPathListDlgBtnPanel let getPathListDlg (i: Form) (li:list>) (brackts:list) let getPathParams let bldPath (f:Form) let bldPathS type PathDlg let getListDlg (i: Form) (l:option list) (o:option unit>)) (t:int) let cliHandlr (req:string) (g:DataGridView) let createAndShowMDIFrm let getDesktopForm let getDskLayoutPanel let dskIconBox icnNm let bldIcnPanel icnNm tx hdlr let getDesktopHandler hdlrNm state let getStatusBar state let addDesktopIcons pnl state let bld() type FldTypes let fldList let getDefaultLbl (t:string):Label let getDefaultTextBox:TextBox let getTaskRangeWidget let getLrgTxtBox let getDatePicker let getDefaultFldPanel let getFldPanel (rng:Control[]) let getPanels acc x let build (li:list) # let getPaginationBar (g:DataGridView) let getToolStripDVList state let getToolStrip state let addGridToolBar (f: Form) (g:DataGridView) state # let getCol (g:DataGridView) (n:int):DataGridViewColumn let AddRows (g:DataGridView) (rows:DataGridViewRow[]) let setColHdrStyle (g: DataGridView) # let PopulateDV_v2 (g: DataGridView) (bindingLi:List) # let freezeBand (g: DataGridView) (num:int) rowOrcol let addTTip (g: DataGridView) (colNum:int) (str:string) : DataGridView # # let PopulateDV (g: DataGridView) let SetupDV (f:Form) (g:DataGridView) let addChkBoxColAt (i:int) let addImgColAt (i:int) let paintCustomImgCell g let getDefDV1 let setColHdrs (g:DataGridView) let getDefDV let bld (owner:Form) let bldGrid (owner:Form) let diffOb # let getDiffsLineMode (bef:string) (aft:string) # let getDiffs (bef:string) (aft:string) let getDiffsHtml (bef:string) (aft:string) let getPatch (bef:string) (aft:string) (benc:bool) let applyPatch patch (bef:string) let testDiff (bef:string option) (aft:string option) let setupMainFrm (frm:Form) # # let main argv # # # # # module Init let prefixes let colLi let numLi let txt1Li let txt2Li let setInitialEnv (m:Mtpl<'m,'t,'p>) : Mtpl<'r,'e,'s> let runInitSetup () # let applyShuffled3 li let pickR() let pickR3 let rndChk() # # let btnHandler let initForm let finalSt let mainSt let inp : Mtpl let statState let inp : Mtpl # # let initF () let defFrm :Form let main argv # let mainCE() let main argv #
# module Base let Cmp() module Comm.WWW # let listItr str l let getNthLitm (lrefs:list) (lget:list) let getURL1 url let fetchUrl callback url let myCallback (reader:IO.StreamReader) url let getURL (u:string):string let google() let getHeaders let uri let client let content let AsyncFormPoster() # module Crypto.Base let getPtr() : IntPtr let iterationCount let getHdr r sl iv dat let AESEnc (r:string option) (dat:string) : string let AESDec (b:string) (p:IntPtr option) let genVlt() # let testFetch (rtb:RichTextBox) (frm:Form) # let testVlt() let RSAEncr (inDat:byte[]) (rsaKey:RSAParameters) (oaepPad:bool) let RSADecr (inDat:byte[]) (rsaKey:RSAParameters) (oaepPad:bool) # let cspp let EncrFolder let DecrFolder let SrcFolder let PubKeyFile let keyName let CreateAsmKeys let EncryptFile rsa inFile let DecryptFile rsa inFile let ExportPublicKey let ImportPublicKey let GetPrivateKey_Click(object sender, EventArgs e) # module PathBuilder let hr() let a2z let z2a let liprAll let lipr let rndUns let lilen (l:list<_>) let lifo let liExc let liGetRnd # let getBrack # # let keyGen inL let postProc aStr let Rnum() let getShuffled3 let extendd li let extended let getExtOutcome (mPath2:list<_>) let scrambled5 (li:string list array) let scrambled3 (li:string list array) let scrambled (li:string list array) let genP inL let buildPath path sz let test1() let bldMask (p1:list<_>) (p2:list<_>) (p3:list<_>) # # module Crypto.Cards type Rank type Suit type Card let desc (c:Card) let getRnum f s let bldHand let tryParse (str:string) let testL let resL let runCards() # module Crypto.GenTags let strIn let initFn (x:int) let prnL x let li1 let genTagsForEa (x:char[]) (y:int) (z:int) : list let genRandoms() let testGenTags() module PgnProc module let hr() let lifo let lim let liprAll let lipr # let strContains let tryParse (str:string) let getMatchesForGrpNums let getMatchesForGrpNms let rec replSingle let reg # # # let rec stripComments (gm:string) :string # let getGames # # let getEaPos # # # let getBdHdrs let getGlyph let સપ્લીટ let inp let allRows let toL let ઇસ_ડિજીટ let rowStrToBd let fen2Rows # let getGroupCounts l let getLastMove (s:string) let spawnShellProcess fn gm mv # let getOutp fn gm mv let runAs l let procGames l let runInitialProc (m:Mtpl<'m,'t,'p>) : Mtpl<'r,'e,'s> let load() let initPgn let runStatePGN() # let someGames let getMvTrips let liToStr let res let resA, resB let toPl let mvs:list<_> let mvsMid let lasStr mvsL let toTrips let fenInps # let fenInps # # module TriviaTrebek let strR let somQns let someQns let r let rFull let l() let categsL let resL module TriviaLaureates type Laureates let tryParseInt let txt let લેન let સપ્લીટ let rows let len let trimUnq let noHdr let toTy let main argv
module CitCollector ; # ; let (|StrOut|_|) (str: string) ; let (|LrbCit|_|) (str: string) ; let (|ALDailyCit|_|) (str: string) ; let (|ParseRegex|_|) regex str ; let parseValWithActivePattern str ; let parseVal (str:string) ; let lrbUrlBase ; let lrbUrlNums ; let lrbExpanded ; let aldUrlBase ; let aldUrlNums ; let aldExpanded ; let runAlDgetter() ; let uri ; let autoParse2 tpl ; let autoParse1 (s:string) ; let breakTo5 (s:string) ; let run() ; # ; module internal DatCreater ; let listRaw() ; let procDat (dat:string list) (u:string) ; let tL ; let toOpt ; let funcMapFld acc (x:Cit option) ; let procRes ; let flattened ; let res, acc ; let collapsedRes ; let procCits() ; # ; module DatImporter ; let tkUpdates ; let inp ; let newTaskImp(s:string) ; let adToSStr (a:AdDoc) ; let newAdImp(s:string) ; let importTasksFromStringDelimFile() ; let importAdFromStringDelimFile(inp:string) ; let ser x ; let getDeSerializedDatFile() ; let runSer() ; let runSerS() ; let listContents() ; let reconcileAdDocs() ; let init() ; let testStateVersion() ; #
type SessionId type Env type Mtpl type DocUNID type OptModID type TgtVer type DocFld type Dat # # type Ad type Task # type Article type Cit type Quad type Doc<'d> type Cfg<'c> type TblDef<'d> # module Core let lim let limi let limf let limfs let limfl let liRemAt let liUpdAt let liprAll let lipr let lilen (l:list<_>) let lifo let liExc let reg let rec replSingle # # let ListUpdateAt let ListRemoveAt # let Optઓર let Strબાઇટ # let (|>|) f x let defEnc:Encoding let currentDate let defDt let now() let currTicks let Htrue let Hfalse let toOb let oToStr (o:option<'t>) let (|?) pred x y let optOr (opt:option<'a>) def let isTuple tuple let merge xs ys let tibbie s let curry f a b let uncurry f (a,b) let tryOf f x let tryOfL f li let tryWith f x let bi let ser x let deSer (str : string) let deSerToList (fn:string):list let (|ParseRegex|_|) regex str let (|BarDelim|_|) (str: string) let (|TabDelim|_|) (str: string) let fCoreVer let platform let T let CJ let ``α`` let ``β`` let ``↠`` let verMap let flattenOb let flat let defFont let bur let tyOf let isTy let hasInt let isEnum let pr let hr() let tibbie let defPadding let anc let doc let b64EncB let b64EncB_SingleLine let b64Enc let b64Enc_SingleLine let b64Dec let bytes2Str let getRnum f s let rndUns let initSession (userNm:option) let tryParseInt (str:string) let tmpEnv let mLTxt let topFrmMenuItm let ctrlGetTag let rec ctrlAddTags let get1 type State let getS let putS s let eval m s let exec m s let empty let modif f s let bind k m type StateBuilder() let ``⍒`` type ContinuationBuilder() let ``⍲`` let ``⍭`` f:('a -> 'r) -> 'r # # let M let L let O let T let W let W1 let B let H let C let R2 let R let F let V let combine let Q1 let Q let Q2 let Q3 let Q4 # let L let B() let W let C() let G() let HoT() let ML x y let ટીન x y z let ટીનtp x y z let કારલોસ let બારી f s t let ભોલે let કરાફે f let ઍસોપ a b let પ્રાઈજ a b c let ગ્રુનડીગ a b # let ThreeKingdoms let Yuan let જિન let NorthernAndSouthernDynasties x y z let સૂઈ let addStr let addStr' let addStr2 let હાન let ટેન્ગ let સૂંગ let મિંગ let ક્વિન્ગ let વરેપ1 x y z let વરેપ2 (x,(a,y)) let વરેપ2a (x,(a,y)) let વચલૂં mid aft x let વરેપ befFn mainFn aftFn x let વરેપ_2 befFn mainFn aftFn x let tkEx :seq module CodeServices let getHeader # let loadBldInstr doc let genPortable doc let loadDocLet let saveDocLet doc # # let uri let proc (str:string) let AsyncMain() # module Control let Counties let tmpPick let aFl # # # # # # # # # # let origHdr let getHdr let bldHdr let rec pick30 let flipCase let alignWithPrior let aProcChk let allMatches let hasBang let Proc
module internal Helpers let imgLi let dCobaltBlue let AirForcefBlue let LimeBlue let PlatinumBlue let MediumDark let Dark let Light let Lighter let Darkest let Linen let SlateGray let Lightest let lightColors let darkColors let allColors let getRandomColor let getRandomLightColor let getRandomDarkColor let defPadding:Padding let defFont:Font let ctrlAddRange (rng:Control[]) (c:Control) : Control let ctrlAddHandler (hdlr:option unit>) (c:Control) : Control let getTopForm let stati s o let anc let doc let tibbie s let tryParseInt type Glyphs let CreateGlyphIcon let વિજય_ની_જમીન let લી_વિજય_ની_જમીન let mLTxt let read s let write x s let run x s let eval x s let modify f let topFrmMenuItm let addDropDn let disItm let disLi let centerOnScreen let ctrlGetTag let rec ctrlAddTags let getAdDoc key let getImg nm let txtBtn nm let getTxtButton nm optF let getImgButton nm optF let getChkBoxBtn:CheckBox let getButton txt imgNm optF :Button let getTSTxtBox nm :ToolStripItem let getTSButton txt imgNm optF :ToolStripItem let getTxtLbl txt let getPwdBox let getMenuItm nm (f:EventArgs -> unit) :ToolStripMenuItem let addCopyPasteDisableHandlers (c: Control) let centerFrm (f:Form) let getDefaultDlgFrm (i:Form) let getTitlePanel str let getLRFlowPanel let getTDFlowPanel let getDefaultDlgBtnPanel (o:option unit>) # let getBtnPanel let addMenu (frm:Form) # # let getDlgFrm:Form let btnPanel let titlePanel s let btnCl (x:int) let txtVal (x:int) let tblPanel let dlg1() let validateFrm (sender:obj) (evtArgs:EventArgs) let validatingForm() let errorProviderForm:Form # # module Dlg let getHalfScreenSz let setupDlg type dlg private (own:Form) as this let z let defPadding let getPlatform let sizeButtonWids let getDlgBase let addTitlePanel let addBtnPanel let પલંગ_તોડ_પાન let getDlgBase (i:Form) (sz: int) let addTitlePanel (t:string) (dlg:Form) let sizeButtonWids (lb:list let addBtnPanel (n:int) (dlg:Form) let getDlgMini (i:Form) (t:string option) let getDlg (i:Form) (t:string option) let get3BtnDlgMini (i:Form) (t:string option) let getDlg75 (i:Form) (t:string option) let getDlg100 (i:Form) (t:string option) let get3BtnDlg (i:Form) (t:string option) let get3BtnDlg75 (i:Form) (t:string option) let get3BtnDlg100 (i:Form) (t:string option) let getDlgMidPanel (singleCtrl:Control option) (d:Form) let getTextDlgPanel (strIn:option) (d:Form) let getComboChoiceDlg (befLi:list) (aftLi:list) (d:Form) let getListDlgPanel (l:option>) (d:Form) let combDlgUpdate (iNm:string) (idxOf:int) (idx:int option) let getCombDlgPanel (l:option>) (d:Form) let getSecDlgPanel let getPathDlgPanel (inL:list) let getTagCloudPanel (tl:list<(string * int)>) let getAllCardsDlgPanel let addCombos (inL:list) (f: Form) (p:TableLayoutPanel) : TableLayoutPanel let getArrowBtnPanel (cB:ComboBox) (lB:ListBox) : FlowLayoutPanel let getPathComboBoxes (inL:list>):Panel let getPathPanel (lB:ComboBox) (brackts:list) let getPathListDlgBtnPanel let getPathListDlg (i: Form) (li:list>) (brackts:list) let getPathParams let bldPath (f:Form) let bldPathS let getCloudDlg (dlg: Form) (l:option>) (t:string) (dlgTyp:int) let listConsumeFn let CombConsumeFn let secDlgConsumeFn let pathConsumeFn let listOrderingConsumeFn let tagDlgConsumeFn let getInpDlg msg (i:Form) let getChoiceDlg msg (i:Form) (l1:list) (l2:list) let getCInputDlg inpTxt (i:Form) let addCombos (inL:list) (f: Form) (p:TableLayoutPanel) : TableLayoutPanel let getArrowBtnPanel (cB:ComboBox) (lB:ListBox) : FlowLayoutPanel let getPathComboBoxes (inL:list>):Panel let getPathPanel (lB:ComboBox) (brackts:list) let getPathListDlgBtnPanel let getPathListDlg (i: Form) (li:list>) (brackts:list) let getPathParams let bldPath (f:Form) let bldPathS type PathDlg let getTxtDlg (i: Form) (strIn:option) (o:option unit>) let get9x9Panel (i:Control) let getTblDlg (i: Form) (li:list) (o:option unit>) let getListDlg (i: Form) (l:option list) (o:option unit>)) (t:int) let cliHandlr (req:string) (g:DataGridView) let createAndShowMDIFrm module Desktop let getDesktopForm let getDskLayoutPanel let dskIconBox icnNm let bldIcnPanel icnNm tx hdlr let getDesktopHandler hdlrNm state let getStatusBar state let addDesktopIcons pnl state let bld() module Form # let colorPicker let fontPicker let getCommDlgPnl (lblTxt:string) (dlg:Form) (h:EventHandler) : Panel let bldCfgForm (owner:Form) # type FldTypes let fldList let getDefaultLbl (t:string):Label let getDefaultTextBox:TextBox let getTaskRangeWidget let getLrgTxtBox let getDatePicker let getDefaultFldPanel let getFldPanel (rng:Control[]) let getPanels acc x let build (li:list) # module Grid # let getPaginationBar (g:DataGridView) let getToolStripDVList state let getToolStrip state let addGridToolBar (f: Form) (g:DataGridView) state # let getCol (g:DataGridView) (n:int):DataGridViewColumn let AddRows (g:DataGridView) (rows:DataGridViewRow[]) let setColHdrStyle (g: DataGridView) # let PopulateDV_v2 (g: DataGridView) (bindingLi:List) # let freezeBand (g: DataGridView) (num:int) rowOrcol let addTTip (g: DataGridView) (colNum:int) (str:string) : DataGridView # # let PopulateDV (g: DataGridView) # # let SetupDV # # # let addChkBoxColAt # let addImgColAt let paintCustomImgCell let getDefDV1 let setColHdrs (g:DataGridView) let getDefDV let bld (owner:Form) let bldGrid (owner:Form) module Ide let diffOb # let getDiffsLineMode (bef:string) (aft:string) # let getDiffs (bef:string) (aft:string) let getDiffsHtml (bef:string) (aft:string) let getPatch (bef:string) (aft:string) (benc:bool) let applyPatch patch let testDiff let setupMainFrm # # # # # # # # # let main argv # # # # # # # # # # let oldRunner:Form # module June_Ide let getCtrlByNm let defDlg let mainFrm:Form # # # # # # # # # # # let main argv #
[let flattenOb ; let flat ; let (|>|) f x ; let defEnc:Encoding ; let currentDate ; let defDt ; let now() ; let toOb ; let removeAt idx li ; fsharpc --nologo --out:main.exe main.fs mono main.exeres: [let flattenOb ; let flat ; let (|>|) f x ; let defEnc:Encoding ; let currentDate ; let defDt ; let now() ; let toOb ; let removeAt idx li ; let updateAt idx newVal li ; let currTicks ; let tk ; let ઓર ; let બાઇટ ; type DocUNID ; type OptModID ; type TgtVer ; type DocFld ; type Dat ; # ; # ; type Ad ; type Task ; type Cit ; type Cfg ; type TblDef ; type Doc ; let tbl ; let cfg1 ; let cfg2 ; let a1 ; let a2 ; let c1 ; let l:Doc list ; let l2 ]
let hr() let toOb let flattenOb let flatten let tickSignif() let ListUpdateAt idx newVal li let ListRemoveAt idx li type Mtpl let ob1 let ob2 let ob3 let ob4 let ob6 let r let s let getRes type State<'Mtpl, 'T> let getS let putS s let eval m s let exec m s let empty let modif f s let bind k m type StateBuilder() let ``⍒`` let tplRun type ContinuationBuilder() let ``⍲`` let ``⍭`` f:('a -> 'r) -> 'r let SumList l let FFst ss let TThd ss let rec SSum ss let Remmer() let Disp() let SumListSt
<
/a>
</
a>
</
a>
